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Preface 



The exponential development of sensor technology and computer power over the last few 
decades, transformed signal processing in an essential tool for a wide range of domains such 
as telecommunications, medicine or chemistry. Signal processing plays nowadays a key role 
in the progress of knowledge, from the discoveries on the universe underlying structure, 
to the recent breakthroughs in the understanding of the sub-atom structure of the matter. 
Internet, GSM, GPS, HDTV technologies are also indebted to the accelerated evolution of 
signal processing methods. Today, a major challenge in this domain is the development of 
fast and efficient algorithms capable of dealing with the huge amount of data provided by the 
modern sensor technology. 



This book intends to provide highlights of the current research in signal processing area, to 
offer a snapshot of the recent advances in this field. This work is mainly destined to researchers 
in the signal processing related areas but it is also accessible to anyone with a scientific 
background desiring to have an up-to-date overview of this domain. The twenty-five chapters 
present methodological advances and recent applications of signal processing algorithms in 
various domains as telecommunications, array processing, biology, cryptography, image and 
speech processing. The methodologies illustrated in this book, such as sparse signal recovery, 
are hot topics in the signal processing community at this moment. 



The editor would like to thank all the authors for their excellent contributions in the different 
areas of signal processing and hopes that this book will be of valuable help to the readers. 



January 2010 

Editor 

Sebastian MIRON 

Centre de Recherche en Automatique de Nancy 

Nancy-Universite, CNRS 
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1. Introduction 

It is well known that the convergence of the adaptive filtering algorithms becomes slow when 
the number of coefficients is very large. However, in many applications, such as digital net- 
work and acoustical echo cancelers, the system being modeled presents sparse impulse re- 
sponse, that is, most of its coefficients have small magnitudes. The classical adaptation ap- 
proaches, such as the least-mean square (LMS) and recursive least squares (RLS) algorithms, 
do not take into account the sparseness characteristics of such systems. 

In order to improve the convergence for these applications, several algorithms have been pro- 
posed recently, which employ individual step-sizes for the updating of the different coeffi- 
cients. The adaptation step-sizes are made larger for the coefficients with larger magnitudes, 
resulting in a faster convergence for the most significant coefficients. Such idea was first in- 
troduced in (Duttweiler, 2000) resulting in the so-called proportionate normalized least mean 
square (PNLMS) algorithm. However, the performance of the PNLMS algorithm for the iden- 
tification of non-sparse impulse response can be very poor, even slower than that of the con- 
ventional LMS algorithm. An improved version of such algorithm, which employs an extra 
parameter to control the amount of proportionality in the step-size normalization, was pro- 
posed in (Benesty & Gay, 2002). 

An observed characteristic of the PNLMS algorithm is a rapid initial convergence, due to the 
fast adaptation speed of the large value coefficients, followed by an expressive performance 
degradation, owing to the small adaptation speed of the small value coefficients. Such be- 
havior is more significant in the modeling of not very sparse impulse responses. In order to 
reduce this problem, the application of a non-linear function to the coefficients in the step-size 
normalization was proposed in (Deng & Doroslovacki, 2006). 

The well-known slow convergence of the gradient algorithms for colored input signals is 
also observed in the proportionate-type NLMS algorithms. Implementations that combine 
the ideas of the PNLMS and transform-domain adaptive algorithms were proposed in (Deng 
& Doroslovacki, 2007) and (Petraglia & Barboza, 2008) for accelerating the convergence for 
colored input signals. 

In this chapter, we give an overview of the most important adaptive algorithms developed 
for the fast identification of systems with sparse impulse responses. The convergence of the 
proposed algorithms are compared through computer simulations for the identification of the 
channel impulse responses in a digital network echo cancellation application. 
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2. Sparse Impulse Response Systems 

Sparse impulse responses are encountered in several applications, such as in acoustic and 
digital network echo cancelers. The adaptive filters employed in the modeling of the unknown 
system in such applications present a small number of coefficients with significant magnitude. 
Figure 1 illustrates the modeling of an unknown system w°, which is assumed to be linear, 
time-invariant and of finite impulse response length (N), by an adaptive filter. The vector 
containing the adaptive filter coefficients is denoted as w(n) = [wo(n) W\(n) • • • w-^_i{n)} T 
and its input vector as x(n) = [x(n) x(n — 1) • • • x(n — N + 1)] T . The adaptive filter output is 
denoted as y(n), the desired response asd(n) and the estimation error as e(n). One of the most 
used adaptation techniques is the normalized least mean-square (NLMS) algorithm, shown in 
Table 1, where (3 is a fixed step-size factor and 5 is a small constant needed in order to avoid 
division by zero. 

As shown in Table 1 for the NLMS algorithm, typical initialization parameters are given for 
all algorithms studied in this chapter. 



x[n] ■■ 



vr 



v[n] 

4- 



w(n) 



e[n] 



yW 



Fig. 1. System identification through adaptive filtering. 



Initialization (typical values) 
8 = 0.01, /3 = 0.25 

w(0)= [ tt? (0) w?i(0) ••■ 
Processing and Adaptation 
For n = 0,1,2,- •• 

x(n) = [ x(n) x(n — 1) 
y(n) = x T 0)w(n) 
e(n) =d(n)-y(n) 



%-i(0) 



x(n-N + l) ] ] 



w(n + l) 
End 



win) + j 



x{n)e{n) 



x T (n)x(n) +5 



Table 1. NLMS Algorithm 



Described in the next sections, adaptive algorithms that take into account the sparseness of the 
unknown system impulse response have been recently developed. The convergence behavior 
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of such algorithms depends on how sparse the modeled impulse response is. A sparseness 
measure of an N-length impulse response w was proposed in (Hoyer, 2004) as 



N 



l w lli 



n-Vn\ Vn\\w\\ 2 



(1) 



where 1 1 w| |/ is the /-norm of the vector w. It should be observed that < £ w < 1/ and that 
£ w = when all elements of w are equal in magnitude (non-sparse impulse response) and 
£ w = 1 when only one element of w is non-zero (the sparsest impulse response). 
In the simulations presented throughout this chapter, the identification of the digital network 
channels of ITU-T Recommendation G.168 (G.168, 2004), by an adaptive filter with N = 512 
coefficients, will be considered. Figures 2(a) and 2(b) show the impulse responses of the most 
and least sparse digital network channel models (gml and gm4, respectively) described in 
(G.168, 2004). Figure 2(c) presents the gm4 channel impulse response with a white noise (uni- 
formly distributed in [-0.05,0.05]) added to it, such as to simulate a non-sparse system. The 
corresponding sparseness measures are f w = 0.8970 for the gml channel, £ w = 0.7253 for the 
gm4 channel and £ w = 0.2153 for the gm4 plus noise channel. 



\^~~ 



Samples 
(b) Model gm4 




Samples 
(c) Model gm4+noise 




Fig. 2. Channel impulse responses: (a) gml, (b) gm4 and (c) gm4+noise. 



3. Proportionate-type NLMS Algorithms 

The proportionate-type NLMS algorithms employ a different step-size for each coefficient, 
such that larger adjustments are applied to the larger coefficients (or active coefficients), re- 
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suiting in faster convergence rate when modeling systems with sparse impulse responses. The 
main algorithms of such family are described next. 

3.1 PNLMS Algorithm 

For an adaptive filter with coefficients W{(n), for 1 < i < N — 1, the proportionate normal- 
ized least mean-square (PNLMS) algorithm is presented in Table 2. In this algorithm, a time- 
varying step-size control matrix T(n), whose elements are roughly proportional to the abso- 
lute values of the corresponding coefficients, is included in the update equation (Duttweiler, 
2000). As a result, the large coefficients at a given iteration get significantly more update en- 
ergy than the small ones. The parameter /3 is a fixed step-size factor, 5 is a small constant 
needed in order to avoid division by zero, and Sp and p are small positive constants which 
are important when all the coefficients are zero (such as in the beginning of the adaptation 
process) or when a coefficient is much smaller than the largest one. 

Initialization (typical values) 
5 V = 5 = 0.01, j8 = 0.25, p = 0.01 

w(0)=[ w? (0) w?i(0) ••• w? N _i(0) ] T = 
Processing and Adaptation 
For n = 0,1,2, ••• 

x(n) = [ x(n) x(n — l) ••• x(n — N + l) ] 

y(n) = x T (n)w(n) 

e{n) = d{n)-y(n) 

7min(n) =pmax{Sp,\w (n)\,--- / \w N _ 1 (n)\} 

For i = 0, 1, • • • , N - 1 

ji(n) = max{7 OTZ - n (n),|w i (n)|} 

End 

For i = 0, 1, • • • , N - 1 

Xi (n) = ^ 

End 

r(n) = diag{£oO),-" ,g N -i(n)} 

t +\ f \ T(n)x(n)e(n) 

w(n + 1) = w(n) + B T/ \ y , \ ; // — 
v ; v ; p x T (n)r(n)x(n) + ^ 

End 

Table 2. PNLMS Algorithm 

Figure 3 displays the experimental MSE evolutions of the PNLMS and NLMS algorithms for 
the three channels of Fig. 2 with white Gaussian noise input. In all experiments a white 
Gaussian measurement noise of variance o% = 10 ~ 6 was added to the desired signal. It can 
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Fig. 3. MSE evolution for the PNLMS and NLMS algorithms for white noise input and chan- 
nels (a) gml, (b) gm4 and (c) gm4+noise. 



be observed in Fig. 3 that the PNLMS algorithm converges much faster than the NLMS algo- 
rithm for the sparse channel gml. However, for the dispersive channel gm4+noise the PNLMS 
behaves much worse than the NLMS. For channel gm4 the PNLMS algorithm presents a fast 
initial convergence, which is significantly reduced after 2000 iterations, becoming slower than 
that of the NLMS algorithm. 

3.2 IPNLMS Algorithm 

In the improved proportionate normalized least mean-square (IPNLMS) algorithm, the indi- 
vidual step-sizes are a compromise between the NLMS and the PNLMS step-sizes, resulting 
in a better convergence for different degrees of sparseness of the impulse response (Benesty 
& Gay, 2002). The IPNLMS algorithm is listed in Table 3. It can be observed that for oc = -1 
the step-size control matrix T(n) reduces to ^1 and hence the IPNLMS and NLMS algorithms 
turn identical. For oc = 1, the elements of T(n) become proportional to the absolute values of 
the coefficients, in which case the IPNLMS and PNLMS algorithms show practically the same 
behavior. A typical value for this parameter is oc = —0.5. 

Figure 4 presents the experimental MSE evolutions of the IPNLMS and NLMS algorithms for 
the three channels of Fig. 2 with white Gaussian noise input. From this figure, it can be ob- 
served that for the sparse channel gml, the IPNLMS algorithm produces similar performance 
as the PNLMS algorithm, that is, significantly better than the NLMS algorithm. For the disper- 
sive channel gm4+noise, the IPNLMS performance is similar to that of the NLMS algorithm, 
not presenting the severe convergence degradation of the PNLMS algorithm. For channel 
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Initialization (typical values) 

8 = 0.01, e = 0.001, /3 = 0.25, oc = -0.5 

w(0)=[ w? (0) w;i(0) ••• «7 N _i(0) ] T = 
Processing and Adaptation 
For n = 0,1,2,- •• 

x(ft) = [ *(n) x(ft-l) ••• x(n-N + l) ] T 

y(n) =x T (ft)w(ft) 

e(n) =d(n) -y(n) 

For i = 0, 1, • • • , N - 1 

a f^ - 1^ 4- ( 1 +^)ki(^)l 

<MW TM f r.r-N-1 I 



2 ^ 2Ep 1 K-WI + e 

End 

T(ft) = diag{g (n),--- ,g N -i(w)} 



w(ft + 1) = w(ft) + j 

End 



r(ft)x(ft)e(ft) 
T (ft)r(ft)xO) + £ 



Table 3. IPNLMS Algorithm 



gm4, the IPNLMS algorithm does not present the performance degradation (after the initial 
convergence period) observed in the PNLMS algorithm; however, there is almost no gain in 
the initial convergence speed when compared to the NLMS algorithm. 

3.3 MPNLMS and SPNLMS Algorithms 

In the w-law improved proportionate normalized least mean-square (MPNLMS) algorithm, 
the step-sizes are optimal in the sense of minimizing the convergence rate (considering white 
noise input signal) (Deng & Doroslovacki, 2006). The resulting algorithm employs a non- 
linear (logarithm) function of the coefficients in the step-size control. A simplified version 
of the MPNLMS, referred to as the segmented PNLMS (SPNLMS) algorithm, also proposed 
in (Deng & Doroslovacki, 2006), employs a segmented linear function in order to reduce its 
computational complexity. These two algorithms are presented in Table 4, where the function 
F(-) is defined as 

F(x) = ln(l + fix) (2) 

for the MPNLMS algorithm and 

„, v f 600x, x < 0.005 /ox 

F « = t 3, x> 0.005 (3) 

for the SPNLMS algorithm (Deng & Doroslovacki, 2006). 

Figure 5 shows the experimental MSE evolutions of the MPNLMS and NLMS algorithms for 

the three channels of Fig. 2 with white Gaussian noise input. From this figure, it can be noticed 
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Fig. 4. MSE evolution for the IPNLMS and NLMS algorithms for white noise input and chan- 
nels (a) gml, (b) gm4 and (c) gm4+noise. 



that the MPNLMS algorithm presents significantly faster convergence, when compared to 
the NLMS, PNLMS and IPNLMS algorithms, mainly for the sparse channels gml and gm4. 
However, for the dispersive channel gm4+noise, its convergence is severely degraded, being 
much slower than that of the NLMS algorithm. 

Figure 6 presents the experimental MSE evolutions of the SPNLMS and NLMS algorithms for 
the three channels of Fig. 2 with white Gaussian noise input. Comparing Figs. 5 and 6, it can 
be verified that the use of the simplified non-linear function does not deteriorate meaningfully 
the performance of the MPNLMS algorithm. 



3.4 Variable-Parameter IMPNLMS Algorithm 

The variable-parameter improved ji-law PNLMS (IMPNLMS) algorithm was proposed in 
(L. Liu & Saiki, 2008). In this algorithm, the channel sparseness measure of Eq. (1) was in- 
corporated into the ji-law PNLMS algorithm in order to improve the adaptation convergence 
for dispersive channels. Since the real channel coefficients are not available, the correspond- 
ing sparseness measure is estimated recursively using the current adaptive filter coefficients. 
The resulting algorithm is summarized in Table 5, where the parameter oc(n), which in the 
improved PNLMS algorithm of Table 4 was a predetermined fixed factor, is made variable 
and related to the estimated impulse response sparseness measure £w(w). In addition, also 
proposed in (L. Liu & Saiki, 2008), was the use of the line segment function 



F(x) 



400x, x < 0.005 

8.511x1+1.96, x> 0.005 



(4) 
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Initialization (typical values) 

S p = S = 0.01, /3 = 0.25, p = 1/N 

w(0)= [ w? (0) w?i(0) ••• %_i(0) ] T = 
Processing and Adaptation 
For n = 0,1,2,- •• 

x(n) = [ x(n) x(n - 1) • • • x(n — N + 1) ] 

y(n) = x T (n)w(n) 

e(n) =d(n)-y(n) 

lmin(n) = pmax{5 p ,F (\w (n)\) r - ,F(|w N _i(n)|)} 

For i = 0, 1, • • • , N - 1 

7,-(n) =max{7 mz - n (n),F(|K; i (n)|)} 

End 

For i = 0, 1, • • • , N - 1 

End 

r(n) = diag{g (n), • • • ,g N _i(n)} 

T(n)x(n)e(n) 



w(n + l) = w(n) +j6- 



'x T (n)r(n)x(n) + ^ 
End 



Table 4. MPNLMS and SPNLMS Algorithms 



with which the steady-state misalignment is decreased in comparison to those of the 
MPNLMS and SPNLMS algorithms (Eqs. (2) and (3)), which place too much emphasis on 
small coefficients. 

Figure 7 presents the experimental MSE evolutions of the IMPNLMS and NLMS algorithms 
for the three channels of Fig. 2 with white Gaussian noise input. The good convergence 
behavior of the IMPNLMS algorithm for the sparse and dispersive channels can be observed 
in this figure. 

4. Wavelet-based proportionate-type NLMS Algorithms 

Although the proportionate-type NLMS algorithms produce better convergence than the 
NLMS algorithm when modeling sparse impulse responses with white noise inputs, they suf- 
fer from the same performance degradation as the NLMS when the excitation signal is colored. 
Figure 8 illustrates the performance of the NLMS, MPNLMS and IMPNLMS algorithms for a 
colored input signal, generated by passing a white Gaussian noise with zero-mean and unit 
variance through the filter with transfer function 

u( \ 0-25^3 ,_. 

H(Z) = 1-1.5Z-1-0.25Z-2 - (5) 
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Fig. 5. MSE evolution for the MPNLMS and NLMS algorithms for white noise input and 
channels (a) gml, (b) gm4 and (c) gm4+noise. 



Such input signal has power spectrum similar to speech signal (Ikeda & Sugiyama, 1994). 
In order to improve the adaptation speed of these algorithms in dispersive channels with 
colored input signals, the use of wavelet transform was proposed independently in (Deng & 
Doroslovacki, 2007) and (Petraglia & Barboza, 2008). The resulting algorithms are described 
next. 



4.1 Wavelet-based MPNLMS algorithm (Transform-Domain) 

The transform-domain proportionate technique presented in (Deng & Doroslovacki, 2007) 
employs the f/-law PNLMS algorithm in the wavelet-domain. Besides improving the con- 
vergence of the conventional algorithms owing to power normalization of the step-sizes, the 
wavelet-domain approach may be advantageous in the modeling of non-sparse impulse re- 
sponses, since they usually become more sparse in the transformed domain. The resulting 
algorithm, termed as wavelet-based MPNLMS in the transform-domain (WMPNLMS-TD), is 
described in Table 6. The transformation matrix T is formed by the coefficients of the wavelet 
filters, as defined in (Attallah, 2000), the vector z(n) = [zo(ft) • • -z^_i(n)} T = Tx(n) is the 
transformed input vector and Pi(n) is the power estimate of z z (n). The matrix D(n) intro- 
duced in the update equation assigns a different step-size normalization to each coefficient. 

Figure 9 presents the experimental MSE evolutions of the WMPNLMS-TD algorithm for 
the three channels of Fig. 2 and colored noise input with the following wavelet functions: 
Haar, Daubechies 2 (Db2) and Daubechies 4 (Db4). Comparing the simulation results of the 
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Fig. 6. MSE evolution for the SPNLMS and NLMS algorithms for white noise input and chan- 
nels (a) gml, (b) gm4 and (c) gm4+noise. 



WMPNLMS-TD algorithm with those of Fig. 8, it can be observed that, for colored input, 
there is a significant improvement in the performance of the MPNLMS algorithm when it 
is applied in the wavelet-domain. The more selective wavelet (Daubechies 4) produced the 
fastest convergence, as expected. 



4.2 Wavelet-based MPNLMS Algorithm (Sparse Filters) 

The wavelet-based proportionate NLMS algorithm proposed in (Petraglia & Barboza, 2008) 
employs a wavelet transform and sparse adaptive filters. Illustrated in Fig. 10, the wavelet 
transform is represented by a non-uniform filter bank with analysis filters Hfc(z), and sparse 
adaptive subfilters G^z 1 *) (Petraglia & Torres, 2002). For an octave-band wavelet, the equiv- 
alent analysis filters of the M-channel filter bank are (Vaidyanathan, 1993) 

M-2 

h ( Z ) = n«V), 



H k (z) = H 1 (z 2M - 1 - k ) M Y[ 2 H°(z 2 '),k=l,---,M-l, 
;=0 



(6) 
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Initialization (typical values) 

5 = 0.01, e = 0.001, j8 = 0.25, A = 0.1, £(-1) = 0.96 

w(0)=[ H7 (0) wi(0) ••• %_i(0) ] T = o 

Processing and Adaptation 

For n = 0,1,2, ••• 

x(n)=[x(n) x(n-l) ••• x(n-N + l) ] T 

y(n) = x T (n)w(n) 

e{n) =d{n)-y(n) 



? ( - N 



^/ne-Io 1 !^^)! 2 , 



£(n) = (l-A)£(n-l)+Ag; w (n) 

«(n)=2£(n)-l 

For z = 0, 1, • • • , N - 1 

a ^ = ^"M 4. (i + *(*))r(Mn)|) 

ffiU " 2 ^ + 2Ef = - 1 F(K-W|) + e 
End 

r(n) = diag{g (n),-" ,g N -i(w)} 

/ +\ f \ T(n)x(n)e(n) 

w(n + 1) = w(n) + j6 T/ J, \ , V c 

v y r x T 0)rO)x(ft) + £ 



End 



Table 5. Variable-Parameter IMPNLMS Algorithms 

where H° (z) and H 1 (z) are, respectively, the lowpass and high-pass filters associated with the 
wavelet functions (Vaidyanathan, 1993). The sparsity factors are 

L = 2 M -\ L k = 2 M ~ k , fc = l,...,M-l, (7) 

and the delays A^ in Fig. 10, introduced for the purpose of matching the delays of the different 
length analysis filters, are given by A k = Nh — Nn k , where Nn k is the length of the kth 
analysis filter. This structure yields an additional system delay (compared to a direct-form 
FIR structure) equal to Ap = Nh q . For the modeling of a length N FIR system, the number 
of adaptive coefficients of the subfilters G k (z) (non-zero coefficients of G k (z Lk )) should be at 
least 



N k - 



N + N Fk 



(8) 



where Np k are the lengths of the corresponding synthesis filters which, when associated to the 
analysis filters H k (z), lead to perfect reconstruction. 

The resulting proportionate-type NLMS algorithm, referred here as wavelet-based MPNLMS 
with sparse filters (WMPNLMS-SF), is presented in Table 7, where x k (n) is the input signal of 
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Fig. 7. MSE evolution for the IMPNLMS and NLMS algorithms for white noise input and 
channels (a) gml, (b) gm4 and (c) gm4+noise. 



the k-th subband (x(n) filtered by H^(z)) and zv^j is the z-th coefficient of G^z). For colored in- 
put signals, the WMPNLMS-SF algorithm presents faster convergence than the time-domain 
MPNLMS algorithm, since its step-size normalization strategy uses the input power at the 
different frequency bands. It should be observed that the step-size normalization takes into 
account the absolute value of each coefficient in comparison to the values of the corresponding 
subfilter coefficients (and not to all coefficients, as is done in the WMPNLMS-TD algorithm 
(Deng & Doroslovacki, 2007)). As a result, the large coefficients of a given subfilter get sig- 
nificantly more of the update energy assigned to the corresponding subband than the small 
ones. 

Figure 11 shows the experimental MSE evolution of the WMPNLMS-SF algorithm for a 
two-level decomposition (M = 3 subbands) using the following wavelet functions: Haar, 
Daubechies 2, Daubechies 4 and Biorthogonal 4.4 (Bior4.4). With such wavelets, the increase 
in the complexity (compared to the MPNLMS algorithm) and the delay introduced by the de- 
composition are not very large, owing to the reduced orders of the corresponding prototype 
filters. 

From Figs. 8 and 11 we conclude that, for the colored input signal employed in the simula- 
tions, the use of the very simple Haar wavelet transform improves significantly the conver- 
gence rate of the MPNLMS algorithm. The fastest convergence of the WMPNLMS-SF algo- 
rithm was obtained with the more selective Daubechies 4 and Biorthogonal 4.4 wavelets. 
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Fig. 8. MSE evolution for the NLMS, MPNLMS and IMPNLMS algorithms for colored noise 
input and channels (a) gml, (b) gm4 and (c) gm4+noise. 



Comparisons with Figs. 9 and 11 indicate that the step-size normalization strategy adopted by 
the WMPNLMS-SF method is advantageous when compared to that of the WMPNLMS-TD 
method. 

The convergence performance of the WMPNLMS-SF algorithm for non-sparse channels can be 
improved by using the IMPNLMS algorithm independently for each adaptive subfiter. Figure 
12 shows the MSE evolution of the resulting algorithm, referred therein as WIMPNLMS-SF, 
for different wavelets with colored input signal. The improvement in the convergence rate for 
the dispersive channel gm4+noise can be observed by comparing Figs. 11(c) and 12(c). 



5. Conclusions 

In this chapter we presented a family of algorithms developed in the last years for improv- 
ing the convergence of adaptive filters when modeling sparse impulse responses. The per- 
formances of the described techniques, known as proportionate-type LMS algorithms, were 
illustrated through computer simulations in the identification of the digital network channels 
of ITU-T recommendation G.168. The first proposed approach, the PNLMS algorithm, was 
shown to produce fast initial convergence for sparse impulse responses, followed by a sig- 
nificant reduction after the fast initial period. Also, its performance was poor for non-sparse 
impulse responses. Improved versions of the PNLMS algorithm were then described and 
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Initialization (typical values) 
T: wavelet transform matrix 
S p = S = 0.01, /3 = 0.25/N, p = 0.01, a = 0.9 

w(0)= [ w?o(0) w?i(0) ••• %_i(0) ] T = o 
Processing and Adaptation 
For n = 0,1,2,- •• 

x(n) = [ x(n) x(n-l) ••• x(n-N + l)] T 

z(n) = Tx(n) 

y(n) = z (n)w(n) 

e(n) =d(n)-y(n) 

lmin(n) =pmax{6 p ,l?(\wo(n)\) r -- ,F(|w N _i(n)|)} 

For z = 0, 1, • • • , N - 1 

7 f (n) =max{7 min (n),F(|K; i (n)|)} 

End 

For i = 0, 1, • • • , N - 1 

x .( n ) = JM 

Pi(n) = ocpi(n - 1) + (1 - oc) *z}(n) 
End 

D(n) = diag{l/p (n) / . • • ,l/p N _!(n)} 
r(n) =diag{^o(w),-" ,gN-i(«)} 
w(n + 1) = w(n) + ]8D(n)r(n)z(n)e(n) 
End 



Table 6. WMPNLMS-TD Algorithm 



their advantages were verified in the simulation results. Whereas the IPNLMS algorithm pro- 
duced enhanced convergence performance when modeling dispersive impulse responses, the 
MPNLMS employed a non-linear function of the coefficients in the step-size normalization 
in order to obtain optimal convergence rate. The combination of these two techniques and 
the use of a sparseness measure of the impulse response resulted in the variable-parameter 
IMPNLMS algorithm. The fast convergence rate of the proportionate-type algorithms was 
limited to white input signals. In order to extend their performance advantages to colored 
input signals, wavelet-domain algorithms, whose step-size normalization takes into account 
the value of each coefficient as well as the input signal power in the corresponding frequency 
band, were described. Simulations showed that the wavelet-domain PNLMS methods present 
significantly faster convergence rate than do the time-domain PNLMS ones for applications 
in which the system has sparse impulse responses and is excited with colored input signal. 
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Fig. 9. MSE evolution for the WMPNLMS-TD with Haar, Db2 and Db4 wavelets for colored 
noise input and channels (a) gml, (b) gm4 and (c) gm4+noise. 
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Initialization 

S p = 3 = 0.01, £ = 0.25,^ = 0.01 



■ k = 0,lr" ,M-1 


w fc (0)= [ w k/0 (0) w K1 (0) 
1 


>cessing and Adaptation 


• n = 0,1,2,- •• 


For k = 0,l,"' ,M-1 


N„ fc -1 

*fc( n ) = E h k(i)x(n-i 

z=0 



w k,N k 



-i(o) ] T = o 



Xfc(w) = [ Xjt(n) x k (n-L k ) ■■■ x k (n-(N k -l)L k ) ] 
y k (n-A D ) = x[(n)w k (n) 
End 

M-l 

yM = E Vk(n-A D ) 

k=0 

e(n) = d(n — A&) — y(n) 
For k = 0,l,-' ,M-1 

7min,k(n) = pmax{S p/ ¥(\w K0 (n)\) r -- ,F (\w k/Nk (n)\)} 
For z = 0,1, ••• ,N fc _! 

7fc,z( n ) = max {7mm / fc(^)/F (I^mWI)} 

End 

For z' = 0,l, ••• ,N]t_i 

End 

rjfc(n) = diag{g M (n), • • • ,^ /N (n)} 

Tk(n)x k (n)e(n) 



wjfc(n + l) = w fc (n)+£- 



End 
End 



Table 7. WMPNLMS-SF Algorithm 
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Fig. 11. MSE evolution for the WMPNLMS-SF algorithm with Haar, Db2, Db4 and Bior4.4 
wavelets and M = 3, for colored noise input and channels (a) gml, (b) gm4 and (c) gm4+noise. 
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Fig. 12. MSE evolution for the WIMPNLMS-SF algorithm with Haar, Db2, Db4 and Bior4.4 
wavelets and M = 3, for colored noise input and channels (a) gml, (b) gm4 and (c) gm4+noise. 
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1. Introduction 

Array processing techniques aim principally at estimating source Directions Of Arrivals 
(DOA's) based on the observations recorded on a sensor array The vector-sensor technology 
allows the use of polarization as an additional parameter, leading to vector sensor array 
processing. In electromagnetics, a vector sensor is composed of six spatially collocated but 
orthogonally polarized antennas, measuring all six components (three for the electric and 
three for the magnetic fields) of the incident wave. The benefits of considering source 
polarization in signal estimation were illustrated in Burgess and Van Veen (1994); Le Bihan et 
al. (2007); Li (1993); Miron et al. (2006); Nehorai and Paldi (1994); Rahamim et al. (2003); Weiss 
and Friedlander (1993a); Wong and Zoltowski (1997) for diverse signal processing problems. 
Most of these algorithms are based on bilinear polarized source mixture models which suffers 
from identifiability problems. This means that, without any additional constraint, the steering 
vectors of the sources (and implicitly their DOA's) cannot be uniquely determined by matrix 
factorization. The identifiability issues involved in vector sensor applications are investigated 
in Ho et al. (1995); Hochwald and Nehorai (1996); Tan et al. (1996a;b). 

The use of polarization as a third diversity, in addition to the temporal and spatial diversities, 
in vector sensor array processing, leading to a trilinear mixture model, was proposed for the 
first time in Miron et al. (2005). Based on this model, a PARAFAC-based algorithm for signal 
detection, was later introduced in Zhang and Xu (2007). Multilinear models gave rise to a 
great interest in the signal processing community as they exhibit interesting identifiability 
properties; their factorization is unique under mild conditions. Several multilinear algorithms 
were proposed, mainly in telecommunication domain, using different diversity schemes such 
as code diversity Sidiropoulos et al. (2000a), multi-array diversity Sidiropoulos et al. (2000b) 
or time-block diversity Rong et al. (2005). For the trilinear mixture model with polarization 
diversity, we derived in Guo et al. (2008) the identifiability conditions and showed that 
in terms of source separation, the performance of the proposed algorithm is similar to the 
classical non-blind techniques. 
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Nevertheless, the joint estimation of all the three parameters of the sources (DOA, polar- 
ization, and temporal sequence) is time-consuming, and it does not always have a practical 
interest, especially in array processing applications. A novel stochastic algorithm for DOA 
estimation of polarized sources is introduced in this chapter, allowing the estimation of only 
two source parameters (DOA and polarization), and thus presenting a smaller computa- 
tional complexity than its trilinear version Guo et al. (2008). It is based on a quadrilinear 
(fourth-order tensor) representation of the polarized data covariance. The parameters are 
then obtained by CANDECOMP/PARAFAC (CP) decomposition the covariance tensor of the 
polarized data, using a quadrilinear alternating least squares (QALS) approach. A significant 
advantage of the proposed algorithm lies in the fact that the methods based on statistical 
properties of the signals proved to outperform deterministic techniques Swindlehurst et al. 
(1997), provided that the number of samples is sufficiently high. The performance of the 
proposed algorithm is compared in simulations to the trilinear deterministic method, MUSIC 
and ESPRIT for polarized sources and to the Cramer-Rao Bound. 

This chapter is organized as follows. Section 2 provides some multilinear algebra notions, nec- 
essary for the presentation of the multilinear models. In Section 3 we introduce the quadri- 
linear model for the covariance of the polarized data and the identifiability conditions for 
this model are discussed in Section 4. Section 5 presents the QALS algorithm for parameter 
estimation; performance and computational complexity issues are also addressed. Section 6 
compares in simulations the quadrilinear algorithm to its trilinear version Guo et al. (2008), 
to polarized versions of MUSIC Miron et al. (2005) and ESPRIT Zoltowski and Wong (2000b) 
and to the CRB for vector sensor array Nehorai and Paldi (1994). We summarize our findings 
in Section 7. 

2. Multilinear algebra preliminaries 

In multilinear algebra a tensor is a multidimensional array. More formally, an N-way or Nth- 
order tensor is an element of the tensor product of N vector spaces, each of which has its 
own coordinate system. A first-order tensor is a vector, a second-order tensor is matrix and 
tensors of order three or higher are called higher-order tensors. Extending matrix notations to 
multilinear algebra we denote by 

X £ (2 IixIiX '" xIn (1) 

a Nth-order tensor with complex entries. In (1), l\, Ii, • • • , In ar e the dimensions of the N modes 
of X. The entry [i\, ii, . • . , z'n) of X is denoted by x Zl / 2 Zn or by (X ) z - lZ - 2 t j N . For an overview 
of higher-order tensor their applications, the reader is referred to De Lathauwer (1997); Kolda 
and Bader (2007). In this section we restrain ourselves to some basic definitions and elemen- 
tary operations on tensors, necessary for the understanding of the multilinear models and 
algorithms presented in the paper. 

Definition 1 (Norm of a tensor). The norm of a tensor X £ c 7ixI2X '" xIn is the square root of the 
sum of the squares of all its elements, i.e., 



iiani ^ 



*2 in 

This is analogous to the matrix Frobenius norm. 



h h In 

x LL'-LK^n\ 2 - (2) 
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Definition 2 (Outer product). The outer product of two tensors X e £hxi 2 x---xi N and y <G 
C Jixj 2 x-xjp ^ a tensor Xoy e £hx-xl N xhx-xj P de fi nedb y (# o^j^^ = x h .j N y h ^ jp . 

The outer product allows to extend the rank-one matrix definition to tensors. 

Definition 3 (Rank-one tensor). An N-way tensor X £ Qhxl 2 x--xl N j s ran ]^- one [j ft can fr e 
written as the outer product ofN vectors, i.e., 

X = 2l\ o a2 o • • • o sl n (3) 

with a n e C In . 

Tensors can be multiplied together though the notation is much more complex than for ma- 
trices. Here we consider only the multiplication of a tensor by a matrix (or a vector) in mode 
n. 

Definition 4 (n-mode product). The n-mode product of a tensor X £ £hx---xl n x---xl N w j^ a 
matrix U e C^ xI " is denoted by X x n U and is of size l\ x • • • x l n _\ x / x J n+1 x • • • x I N . It is 
defined as : 

In 

(X X n ^)i 1 ...i n _ 1 ji n+1 .J N = 2^ x iii 2 ...iN U ii n - ( 4 ) 

in=l 

Several matrix products are important in multilinear algebra formalism, two of which being 
recalled here. 

Definition 5 (Kronecker product). The Kronecker product of matrices A e C Ix ^ and B £ C KxL , 
denoted by A B, z's a matrix of size IK x JL defined by 



A<g>B 



anB ... aijB 



fl/iB ... fl/jB 



(5) 



Definition 6 (Khatri-Rao product). Given matrices A e (^ X -K and b g C^ xK , i7ze/r Khatri-Rao 
product is a I J x K matrix defined by 

A0B=[ai<g>bi a 2 0b 2 ... a K 0b K ], (6) 

where a^ an^ b^ are the the columns of A and B, respectively. 

A tensor can be also represented into a matrix form, process known as matricization or unfold- 
ing. 

Definition 7 (Matricization). The n-mode matricization of a tensor X £ £hx---xi n x---xi N j s a 

I n x Ii . . . I n _i I n+ i . . . I N size matrix denoted by Xr n y The tensor element (i\, ii, . . . , ijsi) maps to 
matrix element (i n ,j) where 

N fc-1 

; = i+ E (**-i)J* w% j k = ii i m (7) 

This operation is generally used in the alternating least squares algorithms for fitting the CP 
models, as illustrated in section 5.1. 
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3. The quadrilinear model of the data covariance 

We introduce in this section a quadrilinear model for electromagnetic source covariance, 
recorded on a six-component vector sensor array. Suppose the sources are completely po- 
larized and the propagation takes place in an isotropic, homogeneous medium. We start by 
modeling the data measurements under the narrowband assumptions. 

Consider an uniform array of M identical sensors spaced by Ax along the x-axis, collecting 
narrowband signals emitted from K (known a priori) spatially distinct far-field sources. For the 
kth incoming wave, its DOA can be totally determined by the azimuth angle fa £ [0, n) (mea- 
sured from + x-axis) and the elevation angle fa € [— tt/2, tt/2] (measured from the ground) 1 , 
as shown in Fig. 1. 




Fig. 1. 2D-DOA on a vector-sensor array 

On an electromagnetic vector sensor, if the incoming wave has unit power, the electric- and 
magnetic-field measurements in Cartesian coordinates, e(fa, fa, oc^, fa) 

H<Pk,*Pk,Kk,fa 

(1994) 

— sin fa — cos fa sin fa 



a r J*) M Jfixr 



r e V r e z 



and 



(k) (k) (k)-[T 
[hx,hy,hz], can be stacked up in a 6 x 1 vector b^ Nehorai and Paldi 



e{<Pk>ipk>Kk>fo) 
H<Pk,Vk,*k,Pk) 



cos fa 


- cos (p^ sin fa 

- sin fa sin fa 

COSXpk 



- sin fa sin fa 

COStpfc 

sin fa 

— cos fa 





gjfc- 



(8) 



*(Mk) 

The 6x2 matrix F^ = V(fa, fa) is referred to as the steering matrix Nehorai et al. (1999) 
and characterizes the capacity of a vector sensor to convert the information carried on an 
impinging polarized plane wave defined in polar coordinates, into the six electromagnetic- 
field-associated electric signals in the corresponding Cartesian coordinates. A 2 x 1 complex 



1 We assume the sources are all coming from the +y side of the x — z plane. 
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vector 



gfc = gOfc,ft) 



g(p(<Xk,Pk) 
gip(*k,Pk) 



COSCtfc 

— sin % 



sinflfc 

cos% 



COS ft 

j sin ft 



(9) 



is used to depict the polarization state of the kth signal in terms of the orientation angle % £ 
( — tt/2, tt/2] and the ellipticity angle ft G [—zr/4, zr/4] Deschamps (1951). Now we have a 
compact expression b^ = F(<^,tfe)g(A:£,ft) modeling the vector sensor response to the /cth 
polarized source. 

Under the far-field assumption, the spatial response of a M-sensor uniform linear array to the 
kih impinging wave, i.e. the steering vector, presents a Vandermonde structure that can be 
expressed as 

a* = atok^jfc) = [1, fl*, •••,af- 1 ] T , (10) 

where % = exp(;fcoAx cos <pk cos ^) is the inter-sensor phase shift and k$ is the wave number 

of the electromagnetic wave. 

Let p (p = 1, 2, • • • , 6) index the six field components of the vector b^ respectively. Define 



[ai,...,a K ] 



1 






1 

M-l 



X 



(11) 



a M x X matrix containing the spatial responses of the array to the N sources, 

B^[b 1/ ... / b K ] = pFigi,...,F K gK] 
a 6 x K matrix containing the polarization responses and 

s 1 • • • 



A 







sk 



(12) 



(13) 



a X x K diagonal matrix containing the K source signals at some fixed instant. With these 
notations, a snapshot of the output of the array can be organized as a M x 6 matrix 

X = ASB T + N (14) 

with N a M x 6 matrix expressing the noise contribution on the antenna. 
The following assumptions are made 

(Al) Sources are zero-mean, stationary, mutually uncorrelated, ergodic processes 

(A2) The noise is i.i.d. centered, complex Gaussian process of variance a 2 , non-polarized and 
spatially white 

(A3) The sources have distinct DO As 

We define the 4-way covariance of the received data as the M x 6 x M x 6 array 

C xx = E{XoX*} (15) 

where E{.} denotes the mathematical expectation operation. We define also the source covari- 
ance as the K x K x K x K fourth-order tensor 
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C ss =E{SoS*} (16) 

From (14) and (16) and using assumptions (Al) and (A2) the covariance tensor of the received 
data takes the following form 

Cxx = C ss XiAx 2 Bx 3 A* x 4 B* + M (17) 

where A/'isaMx6xMx6 tensor containing the noise power on the sensors. Assumption 
(Al) implies that C$s is a hyperdiagonal tensor (the only non-null entries are those having 
all four indices identical), meaning that Cxx presents a quadrilinear CP structure Harshman 
(1970). The inverse problem for the direct model expressed by (17) is the estimation of matrices 
A and B starting from the 4-way covariance tensor Cxx- 

4. Identifiability of the quadrilinear model 

Before addressing the problem of estimating A and B, the identifiability of the quadrilinear 
model (17) must be studied first. The polarized mixture model (17) is said to be identifiable if 
A and B can be uniquely determined (up to permutation and scaling indeterminacies) from 
Cxx- m multilinear framework Kruskal's condition is a sufficient condition for unique CP 
decomposition, relying on the concept of Kruskal-rank or (/c-rank) Kruskal (1977). 

Definition 8 (k-rank). Given a matrix A e C 7x ^, if every linear combination of I columns has full 
column rank, but this condition does not hold for I + 1, then the k-rank of & is I, written as k\ = I. 

Note that k\ < rank(A) < min(I, /), and both equalities hold when rank(A) = /. 
Kruskal's condition was first introduced in Kruskal (1977) for the three-way arrays and gen- 
eralized later on to multi-way arrays in Sidiropoulos and Bro (2000). We formulate next 
Kruskal's condition for the quadrilinear mixture model expressed by (17), considering the 
noiseless case (A/* in (17) has only zero entries). 

Theorem 1 (Kruskal's condition). Consider the four-way CP model (17). The loading matrices 
A and B can be uniquely estimated (up to column permutation and scaling ambiguities), if but not 
necessarily 

k A + kv + kA*+k B * >2X + 3 (18) 

This implies 

k A + k B >K + 2 (19) 

It was proved Tan et al. (1996a) that in the case of vector sensor arrays, the responses of a 
vector sensor to every three sources of distinct DOA's are linearly independent regardless of 
their polarization states. This means, under the assumption (A3) that k^ > 3. Furthermore, as 
A is a Vandermonde matrix, (A3) also guarantees that k\ = min(M,K). All these results sum 
up into the following corollary: 

Corollary 1. Under the assumptions (A1)-(A3), the DOA's ofK uncorrelated sources can be uniquely 
determined using an M-element vector sensor array if M > K — 1, regardless of the polarization states 
of the incident signals. 

This sufficient condition also sets an upper bound on the minimum number of sensors needed 
to ensure the identifiability of the polarized mixture model. However, the condition M > 
K — 1 is not necessary when considering the polarization states, that is, a lower number of 
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sensors can be used to identify the mixture model, provided that the polarizations of the 
sources are different. Also the symmetry properties of Cxx are not considered and we believe 
that they can be used to obtain milder sufficient conditions for ensuring the identifiability 

5. Source parameters estimation 

We present next the algorithm used for estimating sources DOA's starting from the observa- 
tions on the array and address some issues regarding the accuracy and the complexity of the 
proposed method. 

5.1 Algorithm 

Supposing that L snapshots of the array are recorded and using (Al) an estimate of the polar- 
ized data covariance (15) can be obtained as the temporal sample mean 

£xx = jEm°x*(i). (20) 

For obvious matrix conditioning reasons, the number of snapshots should be greater or equal 
to the number of sensors, i.e. L > K. 

The algorithm proposed in this section includes three sequential steps, during which the 
DOA information is extracted and then refined to yield the final DOA's estimates. These three 
steps are presented next. 



5.1.1 Stepl 

This first step of the algorithm is the estimation of the loading matrices A and B from Cxx- 
This estimation procedure can be accomplished via the Quadrilinear Alternative Least Squares 
(QALS) algorithm Bro (1998), as shown next. 

Denote by C pq = Cxx('-, V' '-> a ) tne {V> <?) tn mat ri* slice (M x M) of the covariance tensor Cxx- 
Also note D p (-) the operator that builds a diagonal matrix from the pth row of another and 
A = diag (E\\si || 2 , . . . , E||sx|| 2 ), the diagonal matrix containing the powers of the sources. The 
matrices A and B can then be determined by minimizing the Least Squares (LS) criterion 

6 2 

(P(a,A,A,B)= £ \\t pq -AAD p (B)D q (B*)A H -(^S pq I M \\ (21) 

V ,q=l 
that equals 



2 



<p(a, A, A,B) = £ \\C pq - AAD p (B)D^(B*)A H | f (22) 






2a 2 £^ft |tr (t pp - AADp(B)D p (B*)A H ) } + 6Ma 4 



V 
where tr(-) computes the trace of a matrix and 9ft(-) denotes the real part of a quantity. 
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<p(<r, A, A,B) = Y, \\Cpq ~ AAD p (B)D^(B*)A H - 2a 2 ^3£ | tr {^w ~ 2MA ) } + 6M(j4 

(23) 
Thus, finding A and B is equivalent to the minimization of (23) with respect to A and B, i.e. 

{A,B} = mina;(A,A,B) (24) 

A,B 

subject to ||flfc|| 2 = Mand \\b k \\ 2 = 2, where 

o;(A, A,B) = £ \t vq - AADp(B)D^(B*)A H | 2 (25) 

The optimization process in (24) can be implemented using QALS algorithm, briefly summa- 
rized as follows. 

Algorithm 1 QALS algorithm for four-way symmetric tensors 

1: INPUT: the estimated data covariance Cxx an d the number of the sources K 
2: Initialize the loading matrices A, B randomly, or using ESPRIT Zoltowski and Wong 
(2000a) for a faster convergence 



Set C = A* and D = B* 
repeat 

"(1)1 

^2)[ 

C = X (3) [(D©AOB) + ] T 

D = X (4) [(A©B©C) + ] T , 



A = X (1) [(B©COD) + ] T 

B = X (2) [(C0D0A)Y 



(4)L 

where (-) + denotes Moore-Penrose pseudoinverse of a matrix 
9: Update C, D by C := (A* + C)/2 and D := (B* + D)/2 
10: until convergence 
11: OUTPUT: estimates of A and B. 

Once the A, B are estimated, the following post-processing is needed for the refined DOA 
estimation. 



5.1.2 Step 2 

The second step of our approach extracts separately the DOA information contained by the 
columns of A (see eq. (10)) and B (see eq. (8)). 

First the estimated matrix B is exploited via the physical relationships between the electric and 
magnetic field given by the Poynting theorem. Recall the Poynting theorem, which reveals the 
mutual orthogonality nature among the three physical quantities related to the kth source: the 
electric field e^, the magnetic field h k/ and the kth source's direction of propagation, i.e., the 
normalized Poynting vector u^. 



Vector sensor array processing for polarized sources 
using a quadrilinear representation of the data covariance 



27 



Ufc 



COS (pfc COS Xpfc 

sin (pi cos ipk 
sinxp^ 



9? 



e fc xh* 
|ejfc||-||hi 



(26) 



Equation (26) gives the cross-product DOA estimator, as suggested in Nehorai and Paldi 
(1994). An estimate of the Poynting vector for the kth source u^- is thus obtained, using the 
previously estimated e^ and b^. 

Secondly, matrix A is used to extract the DOA information embedded in the Vandermonde 

structure of its columns a^. 

Given the noisy steering vector a = [uq a\ • • • %_i] T , its Fourier spectrum is given by 



A(co) 



1 

M 



M-l 

J^ a m exp(-jmcv) 

m=0 



(27) 



as a function of cv. 

Given the Vandermonde structure of the steering vectors, the spectrum magnitude | A(cv) | in 

the absence of noise is maximum for co = coq. In the presence of Gaussian noise, max w | A(to) \ 

provides an maximum likelihood (ML) estimator for cvq = koAxcoscpcosxp as shown in Rife 

and Boorstyn (1974). 

In order to get a more accurate estimator of cvq = koAxcoscpcosxp, we use the following 

processing steps. 

1) We take uniformly Q (Q > M) samples from the spectrum A(co), say {A(2rzq / Q)}^~ , 
and find the coarse estimate d> = 2nq/Q so that A(2nq/Q) has the maximum magni- 
tude. These spectrum samples are identified via the fast Fourier transform (FFT) over 
the zero-padded Q-element sequence {uq, . . . , %_i, 0, . . . , 0}. 

2) Initialized with this coarse estimate, the fine estimate of coq can be sought by maximizing 
|A(o;)|. For example, the quasi-Newton method (see, e.g., Nocedal and Wright (2006)) 

can be used to find the maximizer coq over the local range f q~ , -^4 — - j . 
The normalized phase-shift can then be obtained as q = (/cqAx) -1 arg(o>o). 

5.1.3 Step 3 

In the third step, the two DOA information, obtained at Step 2, are combined in order to 
get a refined estimation of the DOA parameters (p and xp. This step can be formulated as the 
following non-linear optimization problem 



mm 



subject to cos (p cos xp = q. 



(28) 



COS (p cos xp 

sin cp cos xp 

sinxp 

A closed form solution to (28) can be found by transforming it into an alternate problem of 3-D 
geometry, i.e. finding the point on the vertically posed circle cos (p cos xp = q which minimizes 
its Euclidean distance to the point u, as shown in Fig. 2. 

To solve this problem, we do the orthogonal projection of u onto the plane x = Q in the 3-D 
space, then join the perpendicular foot with the center of the circle by a piece of line segment. 
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plane x - 



Fig. 2. Illustration of the geometrical solution to the optimization problem (28). The vector OP represents 
the coarse estimate of Poynting vector u. It is projected orthogonally onto the x = q plane, forming a 
shadow cast O'Q, where O' is the center of the circle of center O on the plane given in the polar coordinates 
as cos cost/? = q. The refined estimate, obtained this way, lies on O'Q. As it is also constrained on the 
circle, it can be sought as their intersection point Q. 



This line segment collides with the circumference of the circle, yielding an intersection point, 

that is the minimizer of the problem. 

Let u = [wi U2 U3] T and define k = u^lu^, then the intersection point is given by 



[<? ±v@ ±i*i 



i-e 2 

1+K 2 



(29) 



where the signs are taken the same as their corresponding entries of vector u. Thus, the az- 
imuth and elevation angles estimates are given by 






arctan 



\Q\ 



V 1+K* 



re — arctan 



\Q\ 



1 + K 2 



if ^ > 
if ^ < 



ip. 



Q 2 + 



1 + K 2 



(30a) 



(30b) 



which completes the DO A estimation procedure. The polarization parameters can be obtained 
in a similar way from B. 

It is noteworthy that this algorithm is not necessarily limited to uniform linear arrays. It can 
be applied to arrays of arbitrary configuration, with minimal modifications. 



5.2 Estimator accuracy and algorithm complexity issues 

This subsection aims at giving some analysis elements on the accuracy and complexity of the 
proposed algorithm (QALS) used for the DOA estimation. 
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An exhaustive and rigorous performance analysis of the proposed algorithm is far from 
being obvious. However, using some simple arguments, we provide elements giving some 
insights into the understanding of the performance of the QALS and allowing to interpret the 
simulation results presented in section 6. 

Cramer-Rao bounds were derived in Liu and Sidiropoulos (2001) for the decomposition of 
multi-ways arrays and in Nehorai and Paldi (1994) for vector sensor arrays. It was shown Liu 
and Sidiropoulos (2001) that higher dimensionality benefits in terms of CRB for a given data 
set. To be specific, consider a data set represented by a four-way CP model. It is obvious that, 
unfolding it along one dimension, it can also be represented by a three-way model. The result 
of Liu and Sidiropoulos (2001) states that than a quadrilinear estimator normally yields better 
performance than a trilinear one. In other word, the use of a four-way ALS on the covariance 
tensor is better sounded that performing a three-way ALS on the unfolded covariance tensor. 
A comparaison can be conducted with respect to the three-way CP estimator used in Guo et 
al. (2008), that will be denoted TALS. The addressed question is the following : is it better to 
perform the trilinear decomposition of the 3-way raw data tensor or the quadriliear decom- 
position of the 4-way convariance tensor ? 
To compare the accuracy of the two algorithms we remind that the variance of an unbiased 

linear estimator of a set of independant parameters is of the order of O ( ^a 2 ) , where P is the 
number of parameters to estimate and N is the number of samples. 

Coming back to the QALS and TALS methods, the main difference between them is that the 
trilinear approach estimates (in addition to A and B), the K temporal sequences of size L. 
More precisely, the number of parameters to estimate equals (6 + M + L)K for the three-way 
approach and (6 + M)K for the quadrilinear method. Nevertheless, TALS is directly applied 
on the three-way raw data, meaning that the number of available observations (samples) is 
6ML while QALS is based on the covariance of the data which, because of the symmetry of the 
covariance tensor, reduces the samples number to half of the entries of £xx> that is 18M 2 . The 
point is that the noise power for the covariance of the data is reduced by the averaging in (20) 
to a 2 /L. If we resume, the estimation variance for TALS is of the order of O ( + 6ml ^ ) 
and of O ( - 18M l j- J for QALS. Let us now analyse the typical situation consisting in having 
a large number of time samples. For large values of L, (L ^> (M + 6)), the variance of TALS 
tends to a constant value O ( ^cr 2 ) while for QALS it tends to 0. This means that QALS 
improves continuously with the sample size while this is not the case for TALS. This analysis 
also applies to the case of MUSIC and ESPRIT since both also work on time averaged data. 



We address next some computational complexity aspects for the two previously discussed 
algorithms. Generally, for an N-way array of size l\ x Ii x • • • x l^ t the complexity of its CP 
decomposition in a sum of K rank-one tensors, using ALS algorithm is O (K Yln=i In) Rajih and 
Comon (2005), for each iteration. Thus, for one iteration, the number of elementary operations 
involved is QALS is of order G(6 2 KM 2 ) and of the order of G(6KML) for TALS. Normally 
6M <C L, meaning that for large data sets QALS should be much faster than its trilinear 
counterpart. In general, the number of iterations required for the decomposition convergence, 
is not determined by the data size only, but is also influenced by the initialisation and the 



30 Signal Processing 



parameter to estimate. This makes an exact theoretical analysis of the algorithms complexity 
rather difficult. Moreover, trilinear factorization algorithms have been extensively studied 
over the last two decades, resulting in improved, fast versions of ALS such as COMFAC 2 , 
while the algorithms for quadrilinear factorizations remained basic. This makes an objective 
comparison of the complexity of the two algorithms even more difficult. 
Compared to MUSIC-like algorithms, which are also based on the estimation of the data co- 
variance, the main advantage of QALS is the identifiability of the model. While MUSIC gen- 
erally needs an exhaustive grid search for the estimation of the source parameters, the quadri- 
linear method yields directly the steering and the polarization vectors for each source. 

6. Simulations and results 

In this section, some typical examples are considered to illustrate the performance of the 
proposed algorithm with respect to different aspects. In all the simulations, we assume the 
inter-element spacing between two adjacent vector sensors is half-wavelength, i.e., Ax = A/2 
and each point on the figures is obtained through R = 500 independent Monte Carlo runs. 
We divided this section into two parts. The first aims at illustrating the efficiency of the novel 
method for the estimation of both DOA parameters (azimuth and elevation angles) and the 
second shows the effects of different parameters on the method. Comparisons are conducted 
to recent high-resolution eigenstructure-based algorithms for polarized sources and to the 
CRB Nehorai and Paldi (1994). 

Example 1: This example is designed to show the efficiency of the proposed algorithm using 
a uniform linear array of vector sensors for the 2D DOA estimation problem. It is compared 
to MUSIC algorithm for polarized sources, presented under different versions in Ferrara and 
Parks (1983); Gong et al. (2009); Miron et al. (2005); Weiss and Friedlander (1993b), to TALS 
Guo et al. (2008) and the Cramer-Rao bound for vector sensor arrays proposed by Nehorai 
Nehorai and Paldi (1994). A number of K = 2 equal power, uncorrelated sources are consid- 
ered. The DOA's are set to be fa = 20°, \pi = 5° for the first source and (p 2 = 30°, xp 2 = 10° 
for the other; the polarization states are oc\ = oc 2 = 45°, /$i = —ft 2 = 15°. In the simula- 
tions, M = 7 sensors are used and in total L = 100 temporal snapshots are available. The 
performance is evaluated in terms of root-mean-square error (RMSE). In the following simu- 
lations we convert the angular RMSE from radians to degrees to make the comparisons more 
intuitive. The performances of these algorithms are shown in Fig. 3(a) and (b) versus the in- 
creasing signal-to-noise ratio (SNR). The SNR is defined per source and per field component 
(6M field components in all). One can observe that all the algorithms present similar per- 
formance and eventually achieve the CRB for high SNR's (above dB in this scenario). At 
low SNR's, nonetheless, our algorithm outperforms MUSIC, presenting a lower SNR thresh- 
old (about 8 dB) for a meaningful estimate. CP methods (TALS and QALS), which are based 
on the LS criterion, are demonstrated to be less sensitive to the noise than MUSIC. This con- 
firms the results presented in Liu and Sidiropoulos (2001) that higher dimension (an increased 
structure of the data) benefits in terms of estimation accuracy. 

Example 2: We examine next the performance of QALS in the presence of four uncorrelated 
sources. For simplicity, we assume all the elevation angles are zero, xp^ = 0° for k = 1, ... ,4, 
and some typical values are chosen for the azimuth angles, respectively: (p\ = 10°, (p 2 = 20°, 



2 COMFAC is a fast implementation of trilinear ALS working with a compressed version of the data 
Sidiropoulos et al. (2000a) 
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(a) RMSE of the DOA estimation for the first source 
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(b) RMSE of the DOA estimation for the second source 
Fig. 3. RMSE of the DOA estimation versus SNR in the presence of two uncorrected sources 
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Fig. 4. RMSE of azimuth angle estimation versus SNR for the second source in the presence of 
four uncorrelated sources 



(pi = 30°, (p\ = 40°. The polarizations parameters are &2 = —45°, fe = — 15° for the second 
source and for the others, the sources have equal orientation and ellipticity angles, 45° and 15° 
respectively. We keep the same configuration of the vector sensor array as in example 1. For 
this example we compare our algorithm to polarized ESPRIT Zoltowski and Wong (2000a;b) 
as well. The following three sets of simulations are designed with respect to the increasing 
value of SNR, number of vector sensors and snapshots. 

Fig. 4 shows the comparison between the four algorithms as the SNR increases. Once again, 
the advantage of the multilinear approaches in tackling DO A problem at low SNR's can be 
observed. The quadrilinear approach seems to perform better than TALS as the SNR increases. 
The MUSIC algorithm is more sensitive to the noise than all the others, yet it reaches the CRB 
as the SNR is high enough. The estimate obtained by ESPRIT is mildly biased. 
Next, we show the effect of the number of vector sensors on the estimators. The SNR is fixed 
to 20 dB and all the other simulation settings are preserved. The results are illustrated on 
Fig. 5. One can see that the DOA's of the four sources can be uniquely identified with only 
two vector sensors (RMSE around 1°), which substantiates our statement on the identifiablity 
of the model in Section 4. As expected, the estimation accuracy is reduced by decreasing the 
number of vector sensors, and the loss becomes important when only few sensors are present 
(four sensors in this case). Again ESPRIT yieds biased estimates. For the trilinear method, 
it is shown that its performance limitation, observed on Fig. 4, can be tackled by using more 
sensors, meaning that the array aperture is a key parameter for TALS. The MUSIC method 
shows mild advantages over the quadrilinear one in the case of few sensors (less than four 
sensors), yet the two yield comparable performance as the number of vector sensors increases 
(superior to the other two methods). 
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Fig. 5. RMSE of azimuth angle estimation versus the number of vector sensors for the second 
source in the presence of four uncorrelated sources 



Finally, we fix the SNR at 20 dB, while keeping the other experimental settings the same as 
in Fig. 4, except for an increasing number of snapshots L which varies from 10 to 1000. Fig. 6 
shows the varying RMSE with respect to the number of snapshots in estimating azimuth an- 
gle of the second source. Once again, the proposed algorithm performs better than TALS. 
Moreover as L becomes important, one can see that TALS tends to a constant value while the 
RMSE for QALS continues to decrease, which confirms the theoretical deductions presented 
in subsection 5.2. 



7. Conclusions 

In this paper we introduced a novel algorithm for DOA estimation for polarized sources, 
based on a four-way PARAFAC representation of the data covariance. A quadrilinear alter- 
nated least squares procedure is used to estimate the steering vectors and the polarization 
vectors of the sources. Compared to MUSIC for polarized sources, the proposed algorithm 
ensures the mixture model identifiability; thus it avoids the exhaustive grid search over the 
parameters space, typical to eigestructure algorithms. An upper bound on the minimum num- 
ber of sensors needed to ensure the identifiability of the mixture model is derived. Given the 
symmetric structure of the data covariance, our algorithm presents a smaller complexity per 
iteration compared to three-way PARAFAC applied directly on the raw data. In terms of 
estimation, the proposed algorithm presents slightly better performance than MUSIC and ES- 
PRIT, thanks to its higher dimensionality and it clearly outperforms the three-way algorithm 
when the number of temporal samples becomes important. The variance of our algorithm 
decreases with an increase in the sample size while for the three-way method it tends asymp- 
totically to a constant value. 



34 



Signal Processing 




Number of snapshots 

Fig. 6. RMSE of azimuth angle estimation versus the number of snapshots for the second 
source in the presence of four uncorrelated sources 



Future works should focus on developing faster algorithms for four-way PARAFAC factor- 
ization in order to take full advantage of the lower complexity of the algorithm. Also, the 
symmetry of the covariance tensor must be taken into account to derive lower bounds on the 
minimum number of sensors needed to ensure the source mixture identifiability 
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1. Abstract 

This book chapter deals with the generation of auditory-inspired spectro-temporal features 
aimed at audio coding. To do so, we first generate sparse audio representations we call 
spikegrams, using projections on gammatone or gammachirp kernels that generate neural 
spikes. Unlike Fourier-based representations, these representations are powerful at identify- 
ing auditory events, such as onsets, offsets, transients and harmonic structures. We show that 
the introduction of adaptiveness in the selection of gammachirp kernels enhances the com- 
pression rate compared to the case where the kernels are non-adaptive. We also integrate a 
masking model that helps reduce bitrate without loss of perceptible audio quality. We then 
quantize coding values using the genetic algorithm that is more optimal than uniform quan- 
tization for this framework. We finally propose a method to extract frequent auditory objects 
(patterns) in the aforementioned sparse representations. The extracted frequency-domain pat- 
terns (auditory objects) help us address spikes (auditory events) collectively rather than indi- 
vidually. When audio compression is needed, the different patterns are stored in a small code- 
book that can be used to efficiently encode audio materials in a lossless way. The approach is 
applied to different audio signals and results are discussed and compared. This work is a first 
step towards the design of a high-quality auditory-inspired " object-based" audio coder. 

2. Introduction 

Non-stationary and time-relative structures such as transients, timing relations among acous- 
tic events, and harmonic periodicities provide important cues for different types of audio 
processing techniques including audio coding, speech recognition, audio localization, and 
auditory scene analysis. Obtaining these cues is a difficult task. The most important reason 
why it is so difficult is that most approaches to signal representation /analysis are block-based, 
i.e. the signal is processed piecewise in a series of discrete blocks. Therefore, transients and 
non-stationary periodicities in the signal can be temporally smeared across blocks. Moreover, 
large changes in the representation of an acoustic event can occur depending on the arbitrary 
alignment of the processing blocks with events in the signal. Signal analysis techniques such 
as windowing or the choice of the transform can reduce these effects, but it would be prefer- 
able if the representation was insensitive to signal shifts. Shift-invariance alone, however, 
is not a sufficient constraint on designing a general sound processing algorithm. A desir- 
able representation should capture the underlying 2D-time-frequency structures, so that they 
are more directly observable and well represented at low bit rates (Smith & Lewicki, 2005). 
These structures must be easily extractable as auditory objects for further processing in cod- 
ing, recognition, etc. 
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The aim of this chapter is to first introduce sparse biologically-inspired coding and then pro- 
pose an auditory-inspired coding scheme, which includes many characteristics of the auditory 
pathway such as sparse coding, masking, auditory object extraction, and recognition (see Fig. 
6). In the next section we will see how sparse codes are generated and why they are efficient. 

3. Sparse Coding 

Research on sparse coding is generally conducted almost independently by two group of re- 
searchers: signal processing engineers and biophysicists. In this chapter, we will try to make 
a link between these two realms. In a mathematical sense, sparse coding generally refers to 
a representation where a small number of components are active. In the biological realm, 
a sparse code generally refers to a representation where a small number of neurons are ac- 
tive with the majority of neurons being inactive or showing low activity (Graham & Field, 
2006). Over the last decade, mathematical explorations into the statistics of natural auditory 
and visual scenes have led to the observation that these scenes, as complex and varied as 
they appear, have an underlying structure that is sparse. Therefore, one can learn a possibly 
overcomplete basis 1 set such that only a small fraction of the basis functions is necessary to 
describe a given audio or video signal. In section 5.1, we will see how these codes can be 
generated by projecting a given signal onto a set of overcomplete kernels. When the cell's 
amplitude is different from zero, we say that the neuron or cell is active and has emitted a 
spike. To show the analogy between sparse 2-D representations and the underlying neural 
activity in the auditory or visual pathway, we call the 2-D sparse representation spikegram (in 
contrast with spectrograms) and the components of a sparse representation cells or neurons 
throughout this chapter. 

In a sparse code, the dimensionality of the analyzed signal is maintained (or even increased). 
However, the number of cells responding to any particular instance of the input signal is min- 
imized. Over the population of likely inputs, every cell has the same probability of producing 
a response but the probability is low for any given cell (Field, 1994). In other words, we have 
a high probability of no response and a high probability of high response, but a reduction 
in the probability of a mid-level response for a given cell. We can thus increase the peaki- 
ness (kurtosis) of the histogram of cell activity and be able to reduce the total number of bits 
(entropy) required to code a given signal in sparse codes by using any known arithmetic cod- 
ing approach. The sparse coding paradigm is in contrast with approaches based on Principal 
Component Analysis (PCA) (or Karhunen-Loeve transform), where the aim is to reduce the 
number of signifcant signal components. Fig. 1 shows the conceptual differences between the 
two approaches as described above. 

Normally, sparseness occurs in space (population sparseness) or in time (lifetime sparseness). 
Population sparseness means that our 2-D sparse representation (spikegram) has very few 
active cells at each instance of time, while lifetime sparseness means that each cell in the 
representation is acitve only for a small fraction of the time span of the audio /video signal. 

3.1 Sparse Coding and ICA 

Sparse coding as described in this chapter can also be related to Independent Component 
Analysis (ICA) (Hyvarinen et al., 2009). In fact for some signals (i.e., an ensemble of natu- 
ral images), the maximization of sparseness for a linear sparse code is basically the same as 



1 A set of bases in which the number of kernels /atoms is higher than the dimension of the audio /video 
signal 
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Fig. 1. Conceptual differences between sparse representations and PCA/Karhunen-Loeve 
transform (reproduced from (Field, 1994)). Note that in some cases the dimensionality of 
the sparse code is even higher than the input signal. 



the maximization of non-gaussianity in the context of overcomplete ICA (Hyvarinen et al., 
2009). Karklin and Lewicki also discussed the limits of applicability of the aforementioned 
equivalence in (Karklin & Lewicki, 2005) (Karklin & Lewicki, 2009). However, in the general 
case where components (cell activities) are not statistically independent (i.e., small patches of 
natural images) and noise is present in the system, maximizing sparseness is not equivalent 
to maximizing non-gaussianity and as a consequence ICA is not equivalent to sparse coding 
anymore. 

4. Advantages of Sparse Coding 

In this section we give some reasons (among others) on why sparse coding is such a powerful 
tool in the processing of audio and video materials. 

Signal-to-Noise Ratio 

A sparse coding scheme can increase the signal-to-noise ratio (Field, 1994). In a sparse code, 
a small subset of cells represents all the variance present in the signal (remember that most of 
the cells are inactive in a sparse code). Therefore, that small active subset must have a high 
response relative to the cells that are inactive (or have outputs equal to zero). Hencce, the 
probability of detecting the correct signal in the presence of noise is increased in the sparse 
coding paradigm compared to the case of a transform (e.g., Fourier Transform) where the 



40 Signal Processing 



variance of the signal is spread more uniformly over all coefficients. It can also be shown that 
sparse /overcomplete coding is optimal when a transmission channel is affected by quantiza- 
tion noise and is of limited capacity (see (Doi et al., 2007) and (Doi & Lewicki, 2005)). 

Correspondence and Feature Detection 

In an ideal sparse code, the activity of any particular basis function has a low probability. 
Since the response of each cell is relatively rare, tasks that require matching of features should 
be more successful, since the search space is only limited to those active cells (Field, 1994). 
It has also be shown that the inclusion of a non-negativeness constraint into the extraction of 
sparse codes can generate representations that are part-based (Pichevar & Rouat, 2008) (Lee 
& Seung, 1999) (Hoyer, 2004). It is presumably easier to find simple parts (primitives) in an 
object than identifying complex shapes. In addition, complex shapes can be charachterized 
by the relationship between parts. Therefore, it seems that non-negative sparse coding can be 
potentially considered as a powerful tool in pattern recognition. 

Storage and Retrieval with Associative Memory 

It has been shown in the literature that when the inputs to an associative memory 2 network 
are sparse, the network can store more patterns and provide more effective retrieval with 
partial information (Field, 1994) (Furber et al., 2007). 

As a simple argument of why sparse codes are efficient for storage and retrieval, Graham 
and Field (Graham & Field, 2006) gave the follwoing example. Consider a collection of 5x5 
pixel images that each contain one block letter of the alphabet. If we looked at the histogram 
of any given pixel, we might discover that the pixel was on roughly half the time. How- 
ever, if we were to represent these letters with templates that respond uniquely to each letter, 
each template would respond just l/26th of the time. This letter code is more sparse-and 
more efficient-relative to a pixel code. Although no information is lost, the letter code would 
produce the lowest information rate. Moreover, a representation that was letter-based (and 
sparse) would provide a more efficient means of learning about the association between let- 
ters. If the associations were between individiual pixels, a relatively complex set of statistical 
relationships would be required to describe the co-occurences of letters (e.g., between the Q 
and U). Sparseness can assit in learning since each unit is providing a relatively complete 
representation of the local structure. 

Shift Invariance 

In transform-based (block-based) coding (e.g., Fourier Transforms), representations are sen- 
sitive to the arbitrary alignment of the blocks (analysis window) (see Fig. 2). Even wavelets 
are shift variant with respect to dilations of the input signal, and in two dimensons, rotations 
of the input signal (Simoncelli et al., 1992). However, with sparse coding techniques as de- 
fined in this manuscript this sensitivity problem is completely solved, since the kernels are 
positioned arbitrarily and independently (Smith & Lewicki, 2005). 

4.1 Physiological evidence for sparse coding 

Much of the discussion in recent years regarding sparse coding has come from the the theoret- 
ical and computational communities but there is substantial physiological evidence for sparse 



2 An associative memory is a dynamical system that saves memory attributes in its state space via attac- 
tors. The idea of associative memory is that when a memory clue is presented, the actual memory that 
is most like the clue will be recapitulated (see (Haykin, 2008) for details). 
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Fig. 2. Block-based representations are sensitive to temporal shifts. The top panel shows a 
speech waveform with two sets of overlaid Hamming windows, Al-3 (continuous lines above 
waveform) and Bl-3(dashed lines below waveform). In the three lower panels, the power 
spectrum (jagged) and Linear Prediction Coding (LPC) spectrum of hamming windows offset 
by <5ms are overlaid (A, continuous; B, dahsed). In either of these, small shifts (e.g., from 
A2 to B2) can lead to large changes in the representation (reproduced from (Smith & Lewicki, 
2005)). 



coding in most biological systems. One neurophysiological theory that predicts the presence 
of sparse codes in the neural system is the efficient coding theory (Barlow, 1961) (Simoncelli & 
Olshausen, 2001). Efficient coding theory states that a sensory system should preserve infor- 
mation about its input while reducing the redundancy of the employed code (Karklin, 2007). 
As stated earlier, an efficient way of reducing redundancy is to make cell activity as sparse as 
possible (both in time and space). On the experimental side, Lennie (Lennie, 2003) estimated 
that given the limited resources of a neuron (i.e., limited energy consumption), the maximum 
number of active neurons is only l/50th of any population of cortical neurons at any given 
time (see also (Baddeley, 1996) for a discussion on the energy efficiency of sparse codes). De- 
Weese and colleagues (DeWeese et al., 2003), recording from auditory neurons in the rat, have 
demonstrated that neurons in Al (a specific cortical area) can reliably produce a signle spike in 
response to a sound. Also, evidence from olfactory systems in insects, somatosensory neurons 
in rat, and recording from rat hippocampus all demonstrate highly sparse responses (Graham 
& Field, 2006). 

Sparse coding in its extreme forms a representation called "grandmother cell" code. In such 
a code, each object in the world (e.g., a grandmother) is represented by a single cell. Some 
evidence from neurophysiology may be linked to the presence of this very hierarchical repre- 
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sentation of information (Afraz et al., 2006). However, this coding scheme does not seem to 
be the prevelant mode of coding in sensory systems. 

Sparse coding prevents accidental conjunction of attributes, which is related to the so-called 
binding problem (Barlow, 1961) (von der Malsburg, 1999) (Wang, 2005) (Pichevar et al., 2006). 
Accidental conjunction is the process in which different features from different stimuli are as- 
sociated together, giving birth to illusions or even hallucinations. Although, sparsely coded 
features are not mutually exclusive, they nonetheless occur infrequently. Therefore, the ac- 
cidental conjunction occurs rarely and not more frequently than in real life where "illusory 
conjunction" (the illusion to associate two different features from different stimuli together) 
occurs rarely. 

5. The Mathematics of Sparse Coding 

In most cases, in order to generate a sparse representation we need to extract an overcomplete 
representation. In an overcomplete representation, the number of basis vectors (kernels) is 
greater than the real dimensionality (number of non-zero eigenvalues in the covariance ma- 
trix of the signal) of the input. In order to generate such overcomplete representations, the 
common approach consists of matching the best kernels to different acoustic cues using dif- 
ferent convergence criteria such as the residual energy. However, the minimization of the 
energy of the residual (error) signal is not sufficient to get an overcomplete representation of 
an input signal. Other constraints such as sparseness must be considered in order to have a 
unique solution. Thus, sparse codes are generated using matching pursuit by matching the 
most optimal kernels to the signal. 

5.1 Generating Overcomplete Representations with Matching Pursuit (MP) 

Matching Pursuit (MP) is a greedy search algorithm (Tropp, 2004) that can be used to extract 
sparse representations over an overcomplete set of kernels. Here is a simple analogy showing 
how MP works. Imagine you want to buy a coffee that costs X units with a limited number 
of coins of higher and lower values. You first pick higher valued coins until you cannot use 
them anymore to cover the differnce between the sum of your picked up coins and X. You then 
switch to lower-valued coins to reach the amount X and continue with smaller and smaller 
coins till either there is no smaller coin left or you reach X units. MP is doing the exact same 
thing in the signal domain. It tries to reconstruct a given signal x(t) by decreasing the energy 
of the atom used to shape the signal at each iteration. In mathematical notations, the signal 
x(t ) can be decomposed into the overcomplete kernels as follow 

M n m 

x(t)=££a™g m (t-Tr)+r x (t), (1) 

ra=l z'=l 

where r?" and af are the temporal position and amplitude of the i-th instance of the kernel 
gm, respectively The notation n m indicates the number of instances of g m , which need not be 
the same across kernels. In addition, the kernels are not restricted in form or length. 
In order to find adequate rf 1 , af, and g m matching pursuit can be used. In this technique the 
signal x(t) is decomposed over a set of kernels so as to capture the structure of the signal. The 
approach consists of iteratively approximating the input signal with successive orthogonal 
projections onto some basis. The signal can be decomposed into 

X(t) =< X{t),g m > gm + r x (t), (2) 
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where < x(i),g m > is the inner product between the signal and the kernel and is equivalent 
to a m in Eq. 1. r x (t) is the residual signal. 

It can be shown (Goodwin & Vetterli, 1999) that the computational load of the matching pur- 
suit can be reduced, if one saves values of all correlations in memory or finds an analytical 
formulation for the correlation given specific kernels. 
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Fig. 3. Spikegram of the harpsichord using the gammatone matching pursuit algorithm (spike 
amplitudes are not represented). Each dot represents the time and the channel where a spike 
is fired. 



5.2 Shape of Kernels 

In the previous section we showed how a signal x(t) can be projected onto a basis of kernels 
g m . The question we address in this section is to find optimal bases for different types of sig- 
nals (e.g., image, audio). As mentioned before, the efficient coding theory states that sensory 
systems might have evolved to highly efficient coding strategies to maximize the information 
conveyed to the brain while minimizing the required energy and neural ressources. This fact 
can be the starting point to finding "optimal waveforms "g m for different sensory signals. 

5.2.1 Best Kernels for Audio 

Smith and Lewicki (Smith & Lewicki, 2006) found the optimal basis g m G G for environmental 
sounds by maximizing the Maximum Likelihood (ML) p(x\G) given that the prior probability 
of a spike, p(s), is sparse. Note that the ML part of the optimization deals with the maximiza- 
tion of the information transfer to the brain and the sparseness prior minimizes the energy 
consumption. Therefore, the optimization here is totally inspired by the efficient coding the- 
ory. In mathematical notation, the kernel functions, g m , are optimized by performing gradient 
ascent on the log data probability (including ML and sparseness terms), 



— logp(x|G) = z-— [logp(x|G,s)+log(p(s))] 



(3) 
If we assume that the noise present in the system is gaussian, Eq. 3 can be rewritten as: 



■E<[f 



(4) 
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where [x — x] T m indicates the residual error over the extent of kernel g m at position t™ and s is 
the estimated s. At the start of the training, Smith and Lewicki initialized g m as Gaussian noise 
and trained (found optimal g m ) by running the optimization on a database of natural sounds. 
The natural sounds ensemble used in training combined a collection of mammalian vocal- 
izations with two classes of environmental sounds: ambient sounds (rustling brush, wind, 
flowing water) and transients (snapping twigs, crunching leaves, impacts of stone or woood). 
Results from optimization show only slight differences between the optimal kernels obtained 
by Eq. 3 and the gammatone/gammachirp (Irino & Patterson, 2006) family of filters that ap- 
proximate cochlea in the inner ear (see Fig. 4). However, as pointed out by Smith and Lewicki, 
totally different kernels will be obtained, if we restrain our training set to only a subclass of 
environmental sound or if we change the type of signal used as the training set. In the re- 
maining of this chapter, we use the safe assumption that the physiologically optimal kernels 
for audio are the gammatone/gammachirp filters. 
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Fig. 4. Efficient coding of a combined sound ensemble consisting of environmental sounds 
and vocalization yields filters similar to the gammatone/gammachirp family. The impulse 
response of some of the optimal filters are shown here (reporduced from (Lewicki, 2002)). 



5.2.2 Best kernels for Image 

By using the same efficient coding theory, and by following the same steps as for extracting the 
optimal basis g m for audio (i.e., optimizing an ML with sparseness prior and Eq. 3), Olshausen 
and Field found that the physiologically optimal kernels for image are Gabor wavelets (Ol- 
shausen & Field, 1996) (see Fig. 5). Since our focus in this chapter is on audio coding, we refer 
the reader to (Olshausen & Field, 1996) (among others) for furhter discussion on the extraction 
of optimal kernels for images. 
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Fig. 5. Results of the search for optimal kernels using maximum likelihood with sparseness 
prior on 12x12 pixel images drawn from natural scenes. The kernels are Gabor-like. Repro- 
duced from (Olshausen & Field, 1996). 



6. A New Paradigm for Audio Coding 

In the second half of this chapter, we will briefly describe the biologically-inspired audio coder 
we have developped based on the concepts already presented in the first half of this chapter 
(i.e., sparse coding). 

6.1 The Bio-Inspired Audio Coder 

The analysis /synthesis part of our universal audio codec is based on the generation of 
auditory-inspired sparse 2-D representations of audio signals, dubbed as spikegrams. The 
spikegrams are generated by projecting the signal onto a set of overcomplete adaptive gam- 
machirp (gammatones with additional tuning parameters) kernels (see section 6.2.2). The 
adaptiveness is a key feature we introduced in Matching Pursuit (MP) to increase the effi- 
ciency of the proposed method (see section 6.2.2). An auditory masking model has been de- 
veloped and integrated into the MP algorithm to extract audible spikes (see section 7). In 
addition a differential encoder of spike parameters based on graph theory is proposed in 
(Pichevar, Najaf-Zadeh, Lahdili & Thibault, 2008). The quantization of the spikes is given 
in section 8. We finally propose a frequent pattern discovery block in section 10. The block 
diagram of all the building blocks of the receiver and transmitter of our proposed universal 
audio coder is depicted in Fig. 6 of which the graph-based optimization of the differential 
encoder is explained in (Pichevar, Najaf-Zadeh, Lahdili & Thibault, 2008). 
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Fig. 6. Block diagram of our proposed Universal Bio-Inspired Audio Coder. 



6.2 Generation of the spike-based representation 

We use here the concept of generating sparse overcomplete representations as described in 
section 5 to design a biologically-inspired sparse audio coder. In section 5.2, we saw that the 
gammatone family of kernels is the optimal class of kernels according to the efficient coding 
theory. Therefore, they are used in our approach. In addition, using asymmetric kernels such 
as gammatone /gammachirp atoms is that they do not create pre-echos at onsets (Goodwin 
& Vetterli, 1999). However, very asymmetric kernels such as damped sinusoids (Goodwin 
& Vetterli, 1999) are not able to model harmonic signals suitably. On the other hand, gam- 
matone /gammachirp kernels have additional parameters that control their attack and decay 
parts (degree of symmetry), which are modified suitably according to the nature of the signal 
in our proposed technique. As described in section 5, the approach used to find the projections 
is an iterative one. In this section, we will compare two variants of the projection technique. 
The first variant, which is non-adaptive, is roughly similar to the general approach used in 
(Smith & Lewicki, 2006), which we applied to the specific task of audio coding. However, we 
proposed the second adaptive variant in (Pichevar et al., 2007), which takes advantage of the 
additional parameters of the gammachirp kernels and the inherent nonlinearity of the audi- 
tory pathway (Irino & Patterson, 2001)(Irino & Patterson, 2006). Some details on each variant 
are given below. 

6.2.1 Non-Adaptive Paradigm 

In the non-adaptive paradigm, only gammatone filters are used. The impulse response of a 
gammatone filter is given by 



g(fct) 



t 3 e -2nbt 



cos(2nf c t) t > 0, 



(5) 
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where f c is the center frequency of the filter, distributed on Equal Rectangular Bandwith (ERB) 
scales. At each step (iteration), the signal is projected onto the gammatone kernels (with dif- 
ferent center frequencies and different time delays). The center frequency and time delay that 
give the maximum projection are chosen and a spike with the value of the projection is added 
to the "auditory representation" at the corresponding center frequency and time delay (see 
Fig. 3). The signal is decomposed into the projections on gammatone kernels plus a residual 
signal r x (t) (see Eqs. 1 and 2). 

6.2.2 Adaptive Paradigm 

In the adaptive paradigm, gammachirp filters are used. The impulse response of a gam- 
machirp filter with the corresponding tuning parameters {b,l,c) is given below 

g(f c , t, b, I, c) = t l - l e- lnht cos(2nf c t + c In*) t > 0. (6) 

It has been shown that the gammachirp filters minimize the scale /time uncertainty (Irino & 
Patterson, 2001). In this approach the chirp factor c, I, and b are found adaptively at each step. 
The chirp factor c allows us to slightly modify the instantaneous frequency of the kernels, I 
and b control the attack and decay of the kernels. However, searching the three parameters in 
the parameter space is a very computationally intensive task. Therefore, we use a suboptimal 
search (Gribonval, 2001) in which, we use the same gammatone filters as the ones used in the 
non-adaptive paradigm with values of I and b given in (Irino & Patterson, 2001). This step 
gives us the center frequency and start time (to) of the best gammatone matching filter. We 
also keep the second best frequency (gammatone kernel) and start time. 

Gmaxl = argmax { | < r, g(J, t , b, I, c) > \ } , g e G (7) 

f,to 
Gmaxi = argmax { | < r, g(J, t , b, I, c) > \ } , g e G - G maxl (8) 

M 

For the sake of simplicity, we use / instead of f c in Eqs. 8 to 11. We then use the information 
found in the first step to find c. In other words, we keep only the set of the best two kernels in 
step one, and try to find the best chirp factor given g £ G max \ U G max 2. 

Gmaxc = argmax {\<r,g(f, t , b,l,c)>\}. (9) 

c 

We then use the information found in the second step to find the best b for g £ G maxc in Eq. 
10, and finally find the best I among g e G max i ) in Eq. 11. 

G m axb = argmax {\<r,g(f, t , b,l,c) > | } (10) 

b 

Gmaxl = argmax{\<r,g(f,t Q ,b,l,c)>\}. (11) 

/ 

Therefore, six parameters are extracted in the adaptive technique for the "auditory represen- 
tation": center frequencies, chirp factors (c), time delays, spike amplitudes, b, and /. The last 
two parameters control the attack and the decay slopes of the kernels. Although, there are ad- 
ditional parameters in this second variant, as shown later, the adaptive technique contributes 
to better coding gains. The reason for this is that we need a much smaller number of filters 
(in the filterbank) and a smaller number of iterations to achieve the same SNR, which roughly 
reflects the audio quality. 
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Speech 


Castanet 


Percussion 


Adapt. 


Non-Adapt. 


Adapt. 


Non-Adapt. 


Adapt. 


Non-Adapt. 


Number of spikes 

Spike gain 

Bitrate (bit /sample) 


10492 
0.13N 
1.98 


35208 
0.44N 
3.07 


6510 

0.08N 

1.54 


24580 
0.30N 
3.03 


9430 

0.12N 

1.93 


29370 
0.37N 
2.90 



Table 1. Comparison of the adaptive and non-adaptive schemes for spike generation for three 
different audio signals. The average saving in bitrate over all materials is around 45%. N is 
the signal length (number of samples in the signal). 



6.3 Comparison of Adaptive and Non-Adaptive Paradigms 

In this section we compare the performance of the adaptive and non-adaptive schemes. Re- 
sults and a comparison of the two different schemes in terms of bitrate and number of spikes 
extracted for high quality (scale 4 on ITU-R impairment scale) are given in Table 1. With the 
adaptive scheme, we observe an average drop of 45% in the bitrate compared to the non- 
adaptive approach. The spike gain (decrease in spikes for a given signal of N samples) de- 
creases drastically when the adaptive paradigm is used as well. Fig. 7 compares the adaptive 
to the non-adaptive approach for different numbers of cochlear channels (the number of center 
frequencies used in the gammatone kernels). 
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32-Channel Non-Adaptive 
16-Channel Adaptive 
64-Channel Non-Adaptive 
32-Channel Adaptive 
25-Channel Adaptive 
16-Channel Non-Adaptive 



Iterations 

Fig. 7. Comparison of the adaptive and non-adaptive spike coding schemes of speech for 
different number of channels. In this figure, only the chirp factor is adapted. For the case 
where all three parameters are adapted see Fig. 3 in (Pichevar et al., 2007). 
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7. Masking of Spikes 

In previous sections, we showed how sparse representations based on spikes can be generated 
using our proposed alogrithm based on MP. We also showed how we can increase the perfor- 
mance of our system by shaping adaptively the kernels used in MP to our signal. However, 
in the previous section we generated a sparse signal that is close to the original signal in the 
mean-squared error sense and we ignored the effects of perceptual masking on the generation 
of the signal. In fact, we will see later in this section that some of the spikes generated in 
section 6 are not perceptible by the human ear. To this end, we will first review the basics of 
masking in the auditory system. 

7.1 Fundamentals of Masking 

Auditory masking occurs when the perception of one sound (the maskee) is affected by the 
presence of another sound (the masker). This happens because the original neural activity 
caused by the first signal is reduced by the neural activity of the other sound in the brain. 
Masking can be classified in two distinct categories. In temporal (non-simultaneous masking) 
the masker and the maskee are not present at the same time. In the case of a spikegram, the 
temporal masking is present when two spikes are fired in the same channel (two dots in the 
same horizontal line on the spikegram) and are relatively close in time. On the other hand, 
simultaneous masking happens when two spikes fire at the same time in different channels. 
Fig. 8 outlines the mechanism of different types of masking. The masker (a spike on the 
spikegram) is presented. This masker can potentially mask another spike (or make the pres- 
ence of another spike inaudible) if the latter falls within the pre- or post-synaptic curves with 
an amplitude below the curve or if it is applied simultaneously with a frequency close enough 
to the masker and with the appropriate amplitude. 



Simultaneous 



Post-Masking 




Time after masker appearance (ms) Time after masker removal (ms) 

Fig. 8. Temporal masking of the human ear. Pre-masking occurs prior to masker onset and 
lasts only a few milliseconds; Post-masking may persist for more than 100 milliseconds after 
masker removal (after (Painter & Spanias, 2000)) 
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Audio 


Mean Subj. 


Mean Sub. 


PEAQ Obj. 


PEAQ Obj. 


Material 


Score for MP 


Score for PMP 


Score for MP 


Score for PMP 


Susan Vega 


-0.7500 


-0.9667 


-0.593 


-0.330 


Trumpet 


-0.9667 


-0.4333 


-1.809 


-0.791 


Orchestra 


-0.7667 


-0.4000 


-1.239 


-0.915 


Harpsichord 


-0.5000 


-0.3667 


-1.699 


-0.867 


Bagpipe 


-0.4000 


-0.2000 


-0.765 


-0.502 


Glockenspiel 


-0.2000 


-0.333 


-0.925 


-1.266 


Plucked Strings 


-0.4667 


-0.2667 


-1.050 


-1.050 



Table 2. Mean subjective and objective scores for a few audio files processed with MP and 
PMP. Objective Difference Grade (ODG) are shown in the table for subjective tests. 



7.2 Perceptual Matching Pursuit 

Based on the masking mechanism explained above, we proposed in (Najaf-Zadeh et al., 2008) 
the Perceptual Matching Pursuit, which basically extend MP to the perceptual domain. By 
doing so, only an audible kernel is extracted at each iteration. Moreover, contrary to the 
matching pursuit algorithm, PMP will stop decomposing an audio signal once there is no 
audible part left in the residual. Details of how the pre- and post-masking curves as in Fig. 8 
are extracted for spikegrams as well as the simultaneous masking are given in (Najaf-Zadeh 
et al., 2008). Here, in table 2 we give results on how PMP retains the same audio quality as 
MP. In order to verify the objective scores, we conducted a semi-formal listening test, based 
on the ITU.R BS. 1116 method, to evaluate the quality of the test signals. Six subjects took part 
in a §triple stimulus hidden referencef test and listened to the audio materials (presented in 
Table 2) over the headphone in a quiet room. The CRC SEAQ software was used in the test 
which allowed the listener to seamlessly switch among the three stimuli. In each trial, the 
stimuli §AT was always the reference stimulus known by the subject. Two other stimuli, §BT 
or §CT, were either a hidden reference, identical to §AT, or a synthesized version of the same 
audio material. None of §BT or §CT was known to the subject. The listener had to identify 
the synthesized version (either §BT or §CT) and to grade its quality relative to that of the 
reference on §AT. The grading scale was continuous from 1 (very annoying) to 5 (no difference 
between the reference and the synthesized file). The average subjective scores for MP and 
PMP were 4.4214 and 4.5762, and the standard deviations of the scores were 0.2522 and 0.2612 
respectively. Values in Table 2 are the mapping of the subjective test scores (between 1 and 5) to 
the Objective Difference Grade (ODG) that varies between -4 to +4, according to the ITU.R BS 
1116 standard. Positive values in the ODG represents evaluation errors by subjects (basically 
errors in identifying the hidden reference), while negative values are the subjective scores, 
with being the case where no difference between the reference and the coded material is 
detected by the subject 3 and -4 representing the biggest difference between the reference and 
the coded signal. Although the confidence intervals for the subjective scores are overlapping, 
the majority of the test materials received higher subjective scores for PMP, which is consistent 
with the objective evaluation. The reader may notice that PMP reduces the total number of 
spikes to be extracted for the same audio quality, thus requires lower bitrate for the same 
audio quality. 



' , http://www.itu.int/rec/R-REC-BS.1116-l-199710-I/e 
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8. Quantization of Spikes 

The amplitude of spikes generated in spikegrams should be quantized before transmission. 
The cost function we use to find the optimal levels of quantization is a trade-off between 
the quality of reconstruction and the number of bits required to code each modulus. More 
precisely, given the vector of quantization levels (codebook) q, the cost function to optimize is 
given by (R is the bitrate and D is the distortion): 

£(q) = D + AR = ll^^~^^ll + AH(ff), (12) 

WLiKigi + vV 

where r\ — 10~ 5 , j = 0.001 are set empirically using informal listening tests. The entropy, 
E(q), is computed using the absolute value of spike amplitudes, ff is the vector of quantized 
amplitudes and is computed as follows: 

Oil = qi if qi_i < oti < qi (13) 

H(ff ) is the per spike entropy in bits needed to encode the information content of each element 
of ff defined as: 

H(ft) = -X>(&«)log 2 Ma<), (14) 

i 
where Pi(dci) is the probability density function of ocf. The way the quantizer is defined in 
Eq. 13 reduces the dead zone problem (defined in (Neff & Zakhor, 2000)). To proceed with 
the optimization at a given number of quantization levels, we randomly set the initial values 
(initial population) for the qi and perform Genetic Algorithm to find optimal solutions. The 
goal of the weighting in the denominator of D (Eq. 12) is to give a better reconstruction of 
low-energy parts of the signal. 

Note that in Eq. 12 many different k{ can contribute to the reconstruction of the original sig- 
nal at a given instance of time t, which is not the case when quantization is applied on time 
samples (Lloyd algorithm). Therefore, the optimal dc^ are not statistically independent. In 
addition, in contrast with transform-based coder quantizations (done for instance with Lloyd 
algorithm), gy. are a few atoms selected from a large set of different atoms (tens of thousand) in 
the dictionary and there is an entropy maximization term in our cost function. It is therefore 
impossible to derive a closed-form theoretical solution for the optimal oci in the case of sparse 
representations. Hence, we should use adaptive optimization techniques. In order to avoid 
local minima, in (Pichevar, Najaf-Zadeh, Thibault & Lahdili, 2008) we derived optimal quan- 
tization levels using Genetic Algorithm (GA) (Mitchell, 1998). Results obtained in (Pichevar, 
Najaf-Zadeh, Thibault & Lahdili, 2008) showed that the optimal quantizer is not linear in the 
case of spikegrams. 

9. Piecewise Uniform Quantization 

Running GA for each signal is a time consuming task. In addition, sending a new codebook 
for each signal type and /or frame is an overhead we may want to avoid. In this section we 
propose faster ways to find a suboptimal solution to the quantization results that keeps trans- 
parency in quality. The goal is achieved by performing a piecewise uniform approximation of 
the codebook by using the histogram of the moduli. 

Fig. 9 shows the optimal quantization levels (qi) for four different types of signals. The optimal 
signal is obtained using the GA algorithm explained in the previous section. 
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Fig. 9. Optimal quantization levels for different sound categories. Spike amplitudes are nor- 
malized to one. 



As we can see, the optimal levels can be approximated as piecewise linear segments (meaning 
that the quantizer is "piecewise" linear). The optimal levels are updated by the following 
method for each one-second-long frame: 

• Find the 40-bin histogram h of the spike amplitudes. 

• Threshold the histogram by the sign function so that ht = sign(/z) to find spike ampli- 
tude clusters (concentrations). Smooth out the curves by applying a moving average 
filter with the following impulse response: m(n) = Jjt 0.125S(n — k) for k = 1,2, ...8. 

• Set a crossing threshold of 0.4 on the smoothed curve. Each time the curve crosses the 
threshold, define a new uniform quantizer between the two last threshold crossings. 

9.1 Results from Piecewise Uniform Quantization 

For the different signal types we used in section 2.3, we proceeded with the fast piecewise uni- 
form quantization described in the previous subsection. We noticed that the 32-level quantizer 
gives only near-transparent coding results with CRC-SEAQ (see Table 3) for the piecewise 
uniform quantizer. However, the quality is transparent when 64 levels are used. These ob- 
servations have been confirmed with informal listening tests. This behavior is due to the fact 
that the 64-level quantizer has more uniform quantization levels than the 32-level quantizer. 
Therefore, we recommend the 64-level quantizer when the piecewise uniform approximation 
is used. 

The overall codec bitrate can be computed by combining the bitrate in Tables 1-3 of (Pichevar 
et al., 2007) for the unquantized case and values for amplitude quantization in Table 3 of this 
article. 
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CRC-SEAQ 


32-Levels 


64 Levels 


Percussion 
Castanet 
Harpsichord 
Speech 


-1.30 
-0.50 
-1.10 
-0.95 


-0.25 
-0.10 
-0.15 
-0.44 



Table 3. Comparison of 32-level and 64-level piecewise uniform quantizers for different audio 
signals. A CRC-SEAQ score between and -1 is associated with transparent quality. No 
codebook side information is sent to the receiver in this case. 



10. Extraction of Patterns in Spikegrams 

As mentioned in previous sections, the spike activity of each channel can be associated to 
the activity of a neuron tuned to the center frequency of that channel. The ultimate goal in 
the pattern recognition paradigm is to find a generative neural architecture (such as a synfire 
chain (Abeles, 1991) or a polychronous network (Izhikevich, 2006)) that is able to generate 
a spikegram such as the one we extract by MP (see Fig. 3) for a given audio signal. How- 
ever, for the time being we proposed a solution to a simplified version of the aforementioned 
problem in (Pichevar & Najaf-Zadeh, 2009). In fact, we propose to extract " channel-based or 
frequency-domain patterns" in our generated spikegrams using temporal data mining (Man- 
nila et al., 1997) (Patnaik et al., 2008). Since these patterns are repeated frequently in the signal 
and are the building blocks of the audio signal, we may call them auditory objects (Breg- 
man, 1994). In contrast with other approaches (i.e., Harmonic Matching Pursuit and Meta- 
Molecular Matching Pursuit) that are able to extract some predefined sound structures such 
as harmonics (Krstulovic et al., 2005) very precisely, our proposed approach is able to extract 
patterns without any a priori knowledge of the type of structure present in the sound. The 
reader may also refer to (Karklin, 2007) and (Karklin & Lewicki, 2009) for another approach to 
extract statistical dependencies in a sparse representation that uses latent (hidden) variables 
to exploit higher order statistics. 

10.1 Frequent Episode Discovery 

The frequent episode discovery framework was proposed by Mannila and colleagues (Man- 
nila et al., 1997) and enhanced in (Laxman et al, 2007). Patnaik et al. (Patnaik et al., 2008) 
extended previous results to the processing of neurophysiological data. The frequent episode 
discovery fits in the general paradigm of temporal data mining. The method can be applied to 
either serial episodes (ordered set of events) or to parallel episodes (unordered set of events). 
A frequent episode is one whose frequency exceeds a user specified threshold. Given an 
episode occurrence, we call the largest time difference between any two events constituting 
the occurrence as the span of the occurrence and we use this span as a temporal constraint in 
the algorithm. The details of the algorithm can be found in (Pichevar & Najaf-Zadeh, 2009). 
In (Pichevar & Najaf-Zadeh, 2009), we showed that the extraction of patterns in the spikegram 
is biased towards denser regions. Therefore we proposed a 3-pass extraction algorithm in 
which extracted patterns are subtracted from the original signal at each pass and explore 
sparser regions as well. 

Fig. 10 shows the extracted patterns for each of the three distinct passes for percussion. Since 
unordered episodes are discovered, the order of appearance of spikes in different channels 
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Percussion 


Pass 1 


Pass 2 


Pass 3 


Overall 


No. extracted spikes 


1682 


771 


335 


2788 


No. codebook elements 


47 


36 


11 


94 


Codebook size in bits 


2200 


1976 


320 


4496 


Raw bit saving 


9968 


4403 


1820 


16191 


Effective bit saving 


7768 


2427 


1500 


11695 


Castanet 


Pass 1 


Pass 2 


Pass 3 


Overall 


No. extracted Spikes 


596 


684 


580 


1860 


No. codebook elements 


8 


20 


37 


65 


Codebook size in bits 


440 


1436 


2340 


4216 


Raw bit saving 


2660 


4095 


3253 


10008 


Effective bit saving 


2220 


2659 


913 


5792 


Speech 


Pass 1 


Pass 2 


Pass 3 


Overall 


No. extracted Spikes 


1262 


689 


395 


2346 


No. codebook elements 


8 


21 


11 


40 


Codebook size in bits 


338 


1053 


288 


1679 


Raw bit saving 


3238 


3859 


2250 


11026 


Effective bit saving 


2899 


2806 


1962 


7667 



Table 4. Results for a 3-Pass pattern extraction on 1-second frames. Percussion: The total 
number of bits to address channels when no pattern recognition is used equals 23704 and the 
saving in addressing channels due to our algorithm is 49%. Castanet: The total number of bits 
to address channels when no pattern recognition is used is 21982 and there is a saving of 26% 
with our proposed algorithm. Speech: The total number of bits to address channels when no 
pattern recognition is used is 19118 and there is a saving of 40%. 



can change for a given pattern. However, the channels in which spike activity occurs are the 
same for all similar patterns. In other words the patterns are similar up to a permutation in 
the order of appearance of each spike. Fig. 10 also shows that our 3-pass algorithm is able to 
extract patterns in the high, low and mid-frequency ranges, while a 1-pass algorithm would 
have penalized some sparser spikegram regions. 

In Table 4, the number of extracted spikes is shown for each pass and the raw bit saving 
and effective bit saving in addressing channels as described above are given for percussion, 
Castanet, and speech. Our algorithm was able to extract between 1860 and 2788 spikes in 
different episodes out of the total 4000 spikes. The longest pattern found in percussion is 13- 
spike long and is repeated on average 17 times in the signal frame, while the longest pattern 
for castanet is 14-spike long and is repeated 33 times on average in frames. In the meantime, 
the longest pattern for speech is 100-spike element and is repeated 8 times on average in the 
frames. Results show that the bitrate coding gain obtained in addressing frequency channels 
ranges from 26 % to 49% depending on signal type. Note that since the pattern extraction 
coding is lossless, the informal subjective quality evaluations in (Pichevar et al., 2007) for the 
audio materials still hold when our new audio extraction paradigm is applied. 
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Fig. 10. Spikegrams (dots) and the most relevant extracted patterns (lines) at each of the 3 
passes for percussion for a 250 ms frame. Different colors /grayscales represent different 
episodes. Only spikes not discovered during the previous pass are depicted at each pass. 
Note that since unordered episodes are discovered, patterns are similar up to a permutation 
in the temporal order. 



11. Conclusion 

We outlined in this chapter, a new type of emergent signal coding called sparse represen- 
tation, which is able to preserve the exact timing of acoustical events and edges of images. 
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Sparse codes have also other interesting properties such as shift invariance. Furthermore, we 
discussed the biological plausibiltiy of sparse representations in the brain when it comes to 
the processing of audio and video. We then described our proposed audio coder, which is 
a merger of sparse coding with biologically-inspired coding. We showed how sparse code 
generation (spikegrams), masking, quantization and pattern extraction can be done in our 
proposed framework. Our proposed approach is a first step towards the development of an 
object-based universal audio coder. Object-based coders belong to a totally new generation 
of coders that can potentially reduce current achievable bitrates by an order of magnitude; as 
an analogy, consider the amount of information we can save by transmitting only the color, 
radius, and position of a given circle in a visual scene instead of sending all pixels of that circle 
one by one! 

In a future work, we will extract the structural dependencies of spike amplitudes and /or other 
parameters in the spikegram such as the chirp factor, etc. We will also investigate the design 
of a generative neural model based on spikegrams. Formal subjective listening tests for the 
overall system will be conducted in the future. In order to speed up the spikegram extraction 
of audio signals, we have conducted preliminary tests on replacing the MP stage (see Fig. 
6) by neural circuitry that can be implemented on embedded and parallel hardware. We will 
further explore this avenue in a future work. The application of different ideas outlined in this 
chapter (i.e., pattern recognition and masking model) are not limited to spikegrams and can 
be applied to other sparse representations found in the literature. In addition, the frequency 
episode discovery algorithm discussed in this article can be used in other applications such as 
speech recognition, sound source separation, and audio classification. 
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1. Introduction 

A classical problem is to analyse a signal (function) by decomposing it into suitable building 
blocks, then approximate it by truncating the expansion. Well-known examples are Fourier 
transform and its localized version, the Short Time Fourier transform (sometimes called the 
Gabor transform), and the wavelet transform. In the best case, the elementary blocks form 
a basis in the space of signals, with the pleasant consequence that the expansion coefficients 
are uniquely defined. Unfortunately, this is not always possible and often one has to resort to 
frames. In image processing, in particular, two-dimensional wavelets are by now a standard 
tool in image processing, under the two concurrent approaches, the Discrete Wavelet Trans- 
form (DWT), based on the concept of multiresolution analysis, and the Continuous Wavelet 
Transform (CWT). While the former usually leads to wavelet bases, the CWT has to be dis- 
cretized for numerical implementation and produces in general only frames. 
Nowadays, many situations yield data on spherical surfaces. For instance, in Earth and Space 
sciences (geography, geodesy, meteorology, astronomy, cosmology, etc), in crystallography 
(texture analysis of crystals), in medicine (some organs are regarded as sphere-like surfaces), 
or in computer graphics (modelling of closed surfaces as the graph of a function defined on 
the sphere). So one needs a suitable analysis tool for such data. In the spherical case, the 
Fourier transform amounts to an expansion in spherical harmonics, whose support is the 
whole sphere. Fourier analysis on the sphere is thus global and cumbersome. Therefore many 
different methods have been proposed to replace it with some sort of wavelet analysis. 
In addition, some data may live on more complicated manifolds, such as a two-sheeted hyper- 
boloid, in cosmology for instance (an open expanding model of the universe). In optics also, 
in the catadioptric image processing, where a sensor overlooks a mirror with the shape of a 
hyperboloid or a paraboloid. Another example is a closed sphere-like surface, that is, a surface 
obtained from a sphere by a smooth deformation. Thus it would be useful to have a wavelet 
transform available on such manifolds as well. 

In this chapter, we will review the various aspects of the wavelet transform on the two-sphere, 
both continuous and discrete, with some emphasis on the construction of bases and frames. 
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We will also quickly indicate generalizations to other curved manifolds. Besides the original 
papers, partial reviews of some of this material may be found in (Antoine & Vandergheynst, 
2007; Antoine & Ro§ca, 2008). The present chapter is an elaboration of the paper (Ro§ca & 
Antoine, 2008). 

2. The CWT on the two-sphere 

2.1 Heuristics 

We consider first the extension of the CWT to the two sphere S 2 = {x G R 3 , ||x|| = 1}. 
A complete solution was obtained in (Antoine & Vandergheynst, 1999; Antoine et al., 2002) 
by a group-theoretical method (which actually works in any dimension (Antoine & Van- 
dergheynst, 1998)). As it is well-known in the planar case, the design of a CWT on a given 
manifold X starts by identifying the operations one wants to perform on the finite energy 
signals living on X, that is, functions in L 2 (X,dv), where v is a suitable measure on X. Next 
one realizes these operations by unitary operators on L 2 (X,dv) and one looks for a possible 
group-theoretical derivation. 

In the case of the two-sphere S 2 , the required transformations are of two types: (i) motions, 
which are realized by rotations q G SO(3), and (ii) dilations of some sort by a scale factor 
a G R+. The problem is how to define properly the dilation on the sphere S 2 . The solution 
proposed in (Antoine & Vandergheynst, 1999; Antoine et al., 2002) consists in lifting onto the 
sphere, by inverse stereographic projection, the usual radial dilation in the tangent plane at the 
South Pole. More precisely, one proceeds in three steps: Project a point A e S 2 onto the point 
B in the tangent plane, perform the usual 2-D dilation B h> B fl by a factor a, and project back 
to A a eS 2 . The map A \-^ A a is the stereographic dilation. 

Now, the Hilbert space of spherical signals is L 2 (S 2 ,d}i), where du = sinO d6 dq>, 6 G [0, n] 
is the colatitude angle, <p G [0,2tc) the longitude angle, to = (9,<p) G S 2 . In that space, the 
desired operations are realized by the following unitary operators: 

. rotation R Q : (R Q f)((v) = f{q~ l co), q G SO(3), (1) 

. dilation D a : (D a f)(w) = \{a,0) 1/2 f{cv 1/a ), a G R%. (2) 

In relation (2), oj a := (0 a , cp), 6 a is defined by cot -j- = a cot | for a > and the normalization 
factor (Radon-Nikodym derivative, cocycle) , given as 

A(a,0) 1/2 := la [(a 2 - 1) cosfl + (a 2 + l)]" 1 , (3) 

is needed for compensating the noninvariance under dilation of the natural measure du(to) 
on S 2 . Thus, starting from a function tp G L 2 (S 2 ), we consider the whole family it generates, 
namely, {ip qA := R Q D a xp, q G SO(3), a > 0}. 

By analogy with the plane case, the spherical wavelet transform of a function / G L 2 (S 2 ), with 
respect to the wavelet tp, will be defined as 

*V(<?,«):=(</Vl/>- (4) 

The question, of course, is to determine which functions tp can qualify as wavelets, that is, 
to determine the wavelet admissibility condition. Apart from an educated guess, the natural 
way to find the answer is through a group-theoretical analysis, mimicking the familiar one of 
planar 2-D wavelets. 
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2.2 The group-theoretical or coherent state approach 

As a matter of fact, this spherical CWT was obtained in (Antoine & Vandergheynst, 1999) by 
the group-theoretical approach familiar in the planar 2-D case. The point is to embed the 
rotations from SO(3) and the dilations into the Lorentz group SO (3, 1), the argument being 
that this group is the conformal group both of the sphere S 2 and of the tangent plane IR 2 . The 
embedding results from the so-called Iwasawa decomposition: 

SO (3,l) =SO(3)-A-N, 

where A - SO (l, 1) ~ R ~ R+ (boosts in the z-direction) and N ~ C. 

Then it turns out that the Lorentz group SO (3, 1) has a transitive action on the sphere S 2 . 
In particular, a boost from A corresponds to a stereographic dilation. Now SO (3, 1) has a 
natural unitary representation U in L 2 (S 2 , dji), namely, 

[U(g)f] (co) = A(g,cv) 1/2 f (g^co) , forge SO (3,l), / € L 2 (S 2 ,d F ), (5) 

where \(g,co) = \(a,6) is the Radon-Nikodym derivative (3). 
Thus the parameter space of the spherical CWT is the quotient 

X = SO (3,l)/N~SO(3)-A, 

which is not a group. Therefore, in order to apply the general formalism, we must introduce a 
section a : X — >• SO (3, 1) and consider the reduced representation U{cr{q,a)). Choosing the 
natural (Iwasawa) section cr{q,a) = qa, q G SO(3), a G A, we obtain 

U(a{q,a)) = U(qa) = U(q)U(a) = R Q D a , (6) 

exactly as before, in (l)-(2). 

The following three propositions show that the representation (5) has all the properties that 
are required to generate a useful CWT. First of all, it is square integrable on the quotient man- 
ifold X = SO (3, 1)/N ~ SO(3) • IRf . For simplicity, we shall identify these two isomorphic 
manifolds. 

Proposition 2.1. The UIR (5) is square integrable on X modulo the section a, that is, there exist 
nonzero (admissible) vectors xp G L 2 (S 2 ,d}i) such that 

J -^ J dq \(U{cr{q,a))y\<p)\ 2 := (<p\Ay<p) < oo, for all <p G L 2 (S 2 ,^). (7) 

Here dq is the left invariant (Haar) measure on SO(3). 

The resolution operator (also called frame operator) Ay is diagonal in Fourier space (i.e., it is a 

Fourier multiplier): 

A^f(l,m) = Gy(l)f(l,m), (8) 

where 

G ^ = wh\ £ L S^( Z ' m )| 2 ' foralllef^, (9) 

and xp a (l,m) = (Yj^ltpa), where Yf 1 is a spherical harmonic and ip a := D a xp. 

Next, we have an exact admissibility condition on the wavelets (this condition was also de- 
rived by Holschneider (1996)). 
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Proposition 2.2. An admissible wavelet is a function tp G L 2 (S 2 , du) for which there exists a positive 
constant c < oo such that Gy(l) ^ c, for all I £ N. Equivalently, the function tp e E 2 (S 2 ,dy) is an 
admissible wavelet if and only if the resolution operator Ay is bounded and invertible. 

As in the plane case (Antoine et al., 2004), there is also a weaker admissibility condition on tp: 



I 



s 2 1 — COS I 



■du(tv)=0. (10) 



Here too, this condition is only necessary in general, but it becomes sufficient under mild 
regularity conditions on ip. This is clearly similar to the "zero mean" condition of wavelets on 
the line or the plane. As in the flat case, it implies that the spherical CWT acts as a local filter, 
in the sense that it selects the components of a signal that are similar to tp, which is assumed 
to be well localized. 
Finally, our spherical wavelets generate continuous frames. Indeed: 

Proposition 2.3. For any admissible wavelet tp such that f Q n dcp tp(6, cp) ^ 0, the family {tp a ,Q '= 
R Q D a tp : a > 0, q € SO(3)} is a continuous frame, that is, there exist two constants m > and 
M < oo such that 



< / -3 / d Q \{% /Q \(p)\ 2 ^ M \\(p\\ 2 , for all <j> e L 2 (S 2 ,dy), (11) 

Jo a^ Jso(3) 



da 
Jo a 3 Jso(3) 

or, equivalently, there exist two positive constants d > and c < oo such that 

d^ Gy(l) ^ c, for all I eN 
(in other words, the operators Ay and A^ 1 are both bounded). 

Note that the condition f Q n dcp tp(6,cp) ^ is automatically satisfied for any nonzero axi- 

symmetric (zonal) wavelet. The frame so obtained is not tight, unless Gy(l) = const. For an 

axisymmetric wavelet, tp a (l,m) = tp a (l) is independent of m, hence tightness would require 

that Gy(l) = 8tt 2 Jq°° a~ 3 da \tp a (l) | 2 = const, which seems difficult to obtain. 

With all the ingredients thus available, we may now define the spherical CWT as in (4), 

namely, 

Definition 2.4. Given the admissible wavelet tp, the spherical CWT of a function f <G L 2 (S 2 ,d}i) 
with respect to tp is defined as 

Wyf(q,a) := (tp Q/a \f) = j^[R q D a tp](co)f(co)dy(co). (12) 

As in the planar case, this spherical CWT may be inverted and one gets the following recon- 
struction formula. For / G L 2 (S 2 ) and tp an admissible wavelet such that J Q n tp(6, cp) dcp ^ 0, 
one has 

In addition, the spherical CWT has two important properties: 

(1) It has a correct Euclidean limit. By this we mean that, if we construct the transform on a 

sphere of radius R and then let R —> oo, the spherical CWT tends to the usual planar 2-D 
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CWT on the tangent plane at the South Pole. We refer to (Antoine & Vandergheynst, 1999) for 
mathematical details. 

(2) Unlike the usual 2-D CWT, which is fully covariant with respect to translations, rota- 
tions and dilations, the spherical CWT is only partially covariant. It is covariant under mo- 
tions on S 2 : for any q <G SO(3), the transform of the rotated signal /(q^co) is the function 
]N^f{q~ 1 q,a). But it is not covariant under dilations. Indeed the wavelet transform of the 
dilated signal (D a J)(to) = A(a o ,0) 1/2 /(o; 1 / ao ) is (U(g)xp\f), with g = a~ l qa, and the latter, 
while a well-defined element of SO (3, 1), is not of the form c{q' , a') . This reflects the fact that 
the parameter space X of the spherical CWT is not a group, but only a homogeneous space. 
A byproduct of this analysis is a complete equivalence between the spherical CWT and the 
usual planar CWT in the tangent plane, in the sense that the stereographic projection induces 
a unitary map n : L 2 (S 2 ) — ► L 2 (R 2 ). This fact allows one to lift any plane wavelet, including 
directional ones, onto the sphere by inverse stereographic projection. The same technique will 
be used in Section 3.3 below for lifting the discrete WT onto the sphere and thus generating 
orthogonal wavelet bases on it. 

The advantages of this method are that it is easy to implement (the wavelet tp is given explic- 
itly), it leaves a large freedom in choosing the mother wavelet tp, it allows the use of directional 
wavelets, it preserves smoothness and it gives no distortion around poles, since all points of 
S 2 are equivalent under the action of the operator Rq. However, it is computationally inten- 
sive. As for the disadvantages, the method yields only frames, not bases, as we will see in the 
next section. 

Although this spherical CWT was originally obtained by a group-theoretical method, this 
mathematically sophisticated approach may be short-circuited if one remarks that it is 
uniquely determined by the geometry, in the sense that it suffices to impose conformal behav- 
ior of the relevant maps. More precisely, the stereographic projection is the unique conformal 
diffeomorphism from the sphere to its tangent plane at the South Pole. Similarly, the stereo- 
graphic dilation (2) is the unique longitude-preserving dilation on the sphere that is conformal 
(Wiaux et al., 2005). Thus one gets the formula (12) directly, without the group-theoretical cal- 
culation. 

There is an alternative that also leads to a half-continuous wavelet representation on S 2 . It 
consists in using the so-called harmonic dilation instead of the stereographic one. This dilation 
acts on the Fourier coefficients of a function f, that is, the numbers f^ m := (Y^|/) S 2, where 
{Y™,£ £ N, m = —£,. . .,£} is the orthonormal basis of spherical harmonics in L 2 (S 2 ). The 
dilation d a is defined by the relation 



( d af)i,m := fal,m* a > °- 

This technique, originally due to Holschneider (1996) and Freeden & Windheuser (1997), has 
recently been revived in the applications to astrophysics (Wiaux et al., 2008). However, al- 
though this definition leads to a well-defined, uniquely invertible wavelet representation, 
with steerable wavelets and full rotation invariance, there is no proof so far that it yields a 
frame. Hence one may question the stability of the reconstruction process, since it is the lower 
frame bound that guarantees it. 

2.3 Spherical frames 

The spherical CWT (12) may be discretized and one obtains frames, either half-continuous 
(only the scale variable a is discretized) or fully discrete (Antoine et al., 2002; Bogdanova et al., 
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2005). To be more precise, one gets generalized frames, called weighted frames and controlled 
frames, respectively. They are defined as follows (Jacques, 2004; Bogdanova et al., 2005; Balazs 
et al., 2009). 

Let {(p n : n G T} be a countable family of vectors in a (separable) Hilbert space S) (the index set 
X may be finite or infinite). Then, the family {(p n } is a weighted frame in $) if there are positive 
weights w n and two constants m > and M < oo such that 

m||/|| 2 < E w « I (<Pn\f) | 2 < M||/|| 2 , for all/ 6 Sj. (13) 

The family {<p w } is a controlled frame in f) if there is a positive bounded operator C, with 
bounded inverse, such that 

m Il/H 2 < E <*» I/) </l C 4>»> < M Il/H 2 ' fOT a11 / e «■ d4) 

Clearly this reduces to standard frames for w n = const and C = I, respectively. 
These two notions are in fact mathematically equivalent to the classical notion of frame, 
namely, a family of vectors {cp n } is a weighted frame, resp. a controlled frame, if and only if it 
is a frame in the standard sense (with different frame bounds, of course) (Balazs et al., 2009). 
However, this is not true numerically, the convergence properties of the respective frame ex- 
pansions may be quite different (Antoine et al., 2004; Bogdanova et al., 2005). And, indeed, the 
new notions were introduced precisely for improving the convergence of the reconstruction 
process. 

Following Bogdanova et al. (2005), we first build a half-continuous spherical frame, by dis- 
cretizing the scale variable only, while keeping continuous the position variable on the sphere. 
We choose the half-continuous grid A = {(a;, fly) : to G S 2 , j G Z, a; > fly+i}, where 
A = {aj : j G Z} is an arbitrary decreasing sequence of scales, and Vj := (ay — aj + \)/c8 are 

weights that mimic the natural (Haar) measure da /a 3 . Then a tight frame might be obtained, 
as shown in following proposition. 

Proposition 2.5. Let A = {aj : ; G Z} be a decreasing sequence of scales. Iftp is an axisymmetric 
wavelet for which there exist two constants m, M g 1R+ such that 



where 



m ^ gti>(l) < M, for all I G N, (15) 



then any function f G L 2 (S 2 ,du) may be reconstructed from the corresponding family of spherical 
wavelets, as 

/(") = E v i L d ^') *V(o/,a ; -) f^R^jDa^l (a; 7 ), (16) 

where £y is the (discretized) resolution operator defined by ^ x h(l, m) = gxp{l)~ l h(l, m). 

Note that the resolution operator £y is simply the discretized version of the continuous reso- 
lution operator Ay. Clearly (16) may be interpreted as a (weighted) tight frame controlled by 
the operator tl 1 . 
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Next, still following Bogdanova et al. (2005), one designs a fully discrete spherical frame by 
discretizing all the variables. The scale variable is discretized as before. As for the positions, 
we choose an equiangular grid Qj indexed by the scale level: 

Gj = {co jpq = (0 jp , cp jq ) e S 2 : jp = &£p*, <p jq = f }, (17) 

for p,q e Afj := {n e N : n < 2Bj} and some range of bandwidths B = {Bj e 2N : j e Z}. 
Note that, in (17), the values {0y«} constitute a pseudo-spectral grid, with nodes on the zeros 
of a Chebyshev polynomial of degree 2 By. Their virtue is the existence of an exact quadrature 
rule (Driscoll & Healy, 1994), namely, 



J. 



^M/M = E Wjpfi^jpq)' ( 18 ) 

with certain (explicit) weights Wy^ > and for every band-limited function / 6 L 2 (S 2 ,df/) 
of bandwidth By (i.e., f(l,m) = for all Z > By). Thus the complete discretization grid is 
A(A,B) = {(a jf <v jpq ) :j eZ,p,qe Afj}. 

For this choice of discretization grid, one obtains a discrete weighted, nontight frame, controlled 
by the operator Al 1 , namely, {ipjp q = R[w ]Da$ : j <E Z, p,q e Afj} (Bogdanova et al., 2005): 



m||/|| 2 < E E VjW jp Wyf{cv jpq ,aj)Wyf{cv jpq ,aj) < M ||/|| 2 , (19) 

jeZ p,qeAfj 

where Vy = (ay — fly+i) / 1& are the same positive weights as in Proposition 2.5 and 

%f{q,a) := (Vv|/> = (Aj%D a 1>\f). (20) 

A sufficient condition for (19) to hold may be given, but it is very complicated, involving the 
determinant of an oo-dimensional matrix, unless / is band-limited. As usual, when the frame 
bounds are close enough, approximate reconstruction formulas may be used. The conver- 
gence of the process may still be improved by combining the reconstruction with a conjugate 
gradient algorithm. 

As a matter of fact, no discretization scheme leading to a wavelet basis is known and, in prac- 
tice, the method applies to band-limited functions only. This entails high redundancy and thus 
a higher computing cost, which is not suitable for large data sets. There is also the problem of 
finding an appropriate discretization grid which leads to good frames. Some of them, e.g. the 
equi-angular grid A(A, B) described above, yield exact quadrature rules for the integration of 
band-limited signals on S 2 , but other ones (typically, the familiar HEALPix) are only approxi- 
mate. This is actually a general feature: when discretizing a CWT, it is not easy to prove that 
a given discretization leads to a frame, even less to a good frame or a tight frame. 
For all those reasons, one would prefer to try and build directly a DWT on the sphere. 

3. The DWT on the sphere 

3.1 General requirements 

Many authors have designed methods for constructing discrete spherical wavelets. All of 
them have advantages and drawbacks. These may be characterized in terms of several prop- 
erties which are desirable for any efficient wavelet analysis, planar or spherical (a thorough 
discussion of this topic may be found in Antoine & Ro§ca (2008)). 
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• Basis: The redundancy of frames leads to nonunique expansions. Moreover, the existing 
constructions of spherical frames are sometimes computationally heavy and often applicable 
only to band-limited functions. Thus, in some applications, genuine bases are preferable. 

• Orthogonality: This method leads to orthogonal reconstruction matrices, whose inversion is 
trivial. Thus, orthogonal bases are good for compression, but this is not always sufficient: 
sparsity of reconstruction matrices is still needed in the case of large data sets. 

• Local support: This is crucial when working with large data sets, since it yields sparse ma- 
trices in the implementation of the algorithms. Also, it prevents spreading of " tails" during 
approximation. 1 

• Continuity, smoothness: These properties are always desirable in approximation, but not easily 
achieved. 

3.2 Some known methods 

Let us quote a few of those methods, with focus on the properties just mentioned, without 
being exhaustive. A more comprehensive review, with all references to original papers, may 
be found in (Antoine & Ro§ca, 2008). 

(1) The spherical DWT using spherical harmonics 

Various constructions of discrete spherical wavelets using spherical harmonics may be found 
in the literature, leading to frames or bases. The advantages of this method is that it produces 
no distortion (since no pole has a privileged role) and that it preserves smoothness of the 
wavelets. However, the wavelets so obtained have in general a localized support, but not a 
local one, i.e., it covers the whole sphere. Since this implies full reconstruction matrices, the 
result is not suitable for large amount of data. Examples are the works of Potts et al. (1996) or 
Freeden & Schreiner (1997). 

(2) The spherical DWT via polar coordinates 

The polar coordinate map p : I = [0,/r] x [0,2tt) — > S 2 has the familiar form 

p : (0, cp) \-^ (cos cp sin 0, sin cp sin 0, cos 0) . 

A problem here is continuity. Indeed a continuous function / defined on I remains continuous 
after mapping it onto S 2 if and only if f(6, 0) = f(6, In), for all 6 <G [0, n], and there exists two 
constants P^, Ps such that /(0, cp) = P^ and f(n, cp) = Pg, for all cp G [0,2tt). Unfortunately, 
these continuity conditions are not easily satisfied by wavelets on intervals. 
The obvious advantage of this approach is that many data sets are given in polar coordi- 
nates and thus one does not need to perform additional interpolation when implementing. 
However, there are disadvantages. First, no known construction gives both continuity and 
local support. Next, there are distortions around the poles: p maps the whole segment 
{(0, cp), cp <G [0,2tt)} onto the North Pole, and the whole segment {{n, cp), cp £ [0,2tt)} onto 
the South Pole. Representative examples are papers by Dahlke et al. (1995) or Weinreich (2001). 

(3) The spherical DWT via radial projection from a convex polyhedron 

Let S 2 be the unit sphere centered in and let T be a convex polyhedron, containing in its 
interior and with triangular faces (if some faces are non-triangular, one simply triangularizes 



1 A wavelet has local support if it vanishes identically outside a small region. It is localized if it is negligible 
outside a small region, so that it may have (small, but nonzero) " tails" there. Since these tails may spread 
in the process of approximation of data and spoil their good localization properties, local support is 
definitely preferred (see the example in (Ro§ca & Antoine, 2009)). 
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them). The idea of the method, due to one of us (Ro§ca, 2005; 2007a;b), is to obtain wavelets on 
S 2 first by moving planar wavelets to wavelets defined on the faces of T and then projecting 
these radially onto S 2 . This proceeds as follows. Let Q = dT denote the boundary of Y and let 
p : Q —> S 2 denote the radial projection from the origin: 

p(x,y,z) = p - [x,y,z), where p := p(x,y,z) = 1/ J x 1 + y 2 + z 2 . 

Let T denote the set of triangular faces of T and consider the following weighted scalar prod- 
uct on L 2 (S 2 ): 

(F\G)t= E /, F(0G(0w r (£)^(a ^^fe&jeS 2 , F,GGL 2 (S 2 ). (21) 

Here wr(£i/£2/£3) — 2^ 2 \aj£i + bjfa + C T^3\~ > with aj,bj,cj,dj the coefficients of 
x, y, z, 1, respectively, in the determinant 



x y z 1 

xi y 1 z x 1 

x 2 yi z 2 1 

*3 y?> z 3 l 



: ajx + bjy + cjz + djl, 



where (xj, y z -, z z ), z = 1, 2, 3, are the vertices of the planar triangle T e T. Then one proves that 

the norm IHIr : — (' I ')r ^ s ec l u i va l en t to the usual norm in L 2 (S 2 ), i.e., there exist constants 
m r > 0, M r < oo such that 

mr||/|| r ^ll/ll2^M r ||/|| r ,V/GL 2 (S 2 ). 

Explicit expressions for optimal bounds m^ and Mr are given in (Ro§ca, 2009). 
The resulting wavelets are orthogonal with respect to the weighted scalar product (21) on 
L 2 (S 2 ). This method offers many advantages: no distortion around the poles, possible 
construction of continuous and locally supported stable wavelet bases, local support of the 
wavelets (leading to sparse matrices), easy implementation, possible extension to sphere-like 
surfaces (Ro§ca, 2006). As a disadvantage, we may note the lack of smoothness of the wavelets. 

(4) Needlets 

A new class of discrete spherical wavelets, called needlets, has been introduced recently (Nar- 
cowich et al., 2006a;b; Baldi et al., 2009). These functions, which are actually special spherical 
harmonics kernels, are derived by combining three ideas, namely, a Littlewood-Paley decom- 
position, a suitable distribution of (finitely many) points on the sphere, called centers, and an 
exact quadrature rule. The dilation takes place in the space of spherical harmonics, effectively 
in Fourier space, i.e., it is a harmonic dilation as described at the end of Section 2.2. The 
upshot is a new class of tight frames on the sphere. The frame functions are both compactly 
supported in the frequency domain (i.e., band-limited in /) and almost exponentially localized 
around each center. When combined with a new statistical method, they offer a powerful tool 
for analysing CMB (WMAP) data, e.g. for analysing the cross-correlation between the latter 
and galaxy counts from sky surveys (Pietrobon et al., 2006; Marinucci et al., 2008). They have 
also found nice applications in statistics (Baldi et al., 2008; 2009). 

As a matter of fact, no construction so far has led to wavelet bases on the sphere which are 
simultaneously continuous (or smoother), orthogonal and locally supported, although any two 
of these three conditions may be met at the same time. This suggests to try another approach. 
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3.3 Lifting the DWT from the plane to the sphere 

The method we propose consists in lifting wavelets from the tangent plane to the sphere by in- 
verse stereographic projection (Ro§ca & Antoine, 2009). It yields simultaneously smoothness, 
orthogonality, local support, vanishing moments. The disadvantage is that it gives distortions 
around a pole. In addition, it is not suitable for the whole sphere S 2 , but only for data "away" 
from that pole. However, the latter can be taken anywhere on the sphere, for instance, in a 
region where no data is given. To give an example, European climatologists routinely put the 
North Pole of their spherical grid in the middle of the Pacific Ocean. Therefore, this is in fact 
a minor inconvenient in practice. 
Our sphere is 

S 2 = K = (£i,fc,fo) € R 3 , ({ + Cl + (fe - 1) 2 = 1}, 
where we have used the parametrization Z,\ = cos cp sin 0, £2 = sin cp sin 0, £3 = 1 + cos 6, for 
6 e (0, n], cp e [0,2n). The pointed sphere is S 2 = S 2 \ {(0,0,2)}. 

Let now p : S 2 — >• IR 2 be the stereographic projection from the North Pole N(0,0,2) onto the 
tangent plane £3 = at the South Pole. The area elements dx of IR 2 and f/(£) of S 2 are related 
by dx = v(£) 2 d]i(£), where the weight factor v : S 2 — » IR is defined as 

V(0 = ^ = r ^,f=(f 1 ,fofo)-(^)€S?. 

Notice that L 2 (S 2 ) := L 2 (S 2 ,df/(£)) = L 2 (S 2 ), since the set {N} is of measure zero. As men- 
tioned in Section 2, the stereographic projection p induces a unitary map n : L 2 (S 2 ) — ► L 2 (IR 2 ), 
with inverse zr" 1 : L 2 (IR 2 ) -► L 2 (S 2 ) given by n~ l (F) = v • (F o p), VFG L 2 (IR 2 ). As a con- 
sequence, we have 

(F|G) L2(R2) = (v.(Fo P )|v.(Go P )) L2( g 2) , VF,GGL 2 (R 2 ). (22) 

This equality allows us to construct orthogonal bases on L 2 (S 2 ) starting from orthogonal bases 
in L 2 (IR 2 ) . More precisely, we will use the fact that, if the functions F,G £ L 2 (IR 2 ) are orthogo- 
nal, then the functions F s = v • (F o p) and G s = v • (G o p) will be orthogonal in L 2 (S 2 ). Thus, 
the construction of multiresolution analysis (MRA) and wavelet bases in L 2 (S 2 ) is based on 
the equality (22). 

The starting point is a MRA in L 2 (IR 2 ) (for a thorough analysis of MRAs in 1-D and in 2-D, we 
refer to the monograph (Daubechies, 1992)). For simplicity, we consider 2-D tensor wavelets, 
that is, we take the tensor product of two 1-D MRAs, with scaling function (p, mother wavelet 
ip, and diagonal dilation matrix D = diag(2,2). Thus we get a 2-D MRA of L 2 (IR 2 ), i.e., an 
increasing sequence of closed subspaces Vy C L 2 (IR 2 ) with H/eZ Vy = {0} and U/ez Vy = 
L 2 (IR 2 ), satisfying the following conditions: 

(1) /(•) e Vj ^ f(D •) e v /+1 , 

(2) There exists a function G L 2 (IR 2 ) such that the set {0(- — k), k G Z 2 } is an orthonor- 
mal basis (o.n.b.) of Vq. 

In terms of the original 1-D MRA, the 2-D scaling function is O(x) = (p(x)cp(y) and for the 2-D 
MRA it generates, one has 



v /+l 


= Vy+i V j+1 


= (v-ew y ) 


(8)(T/y©Wy) 






= (Vj®Vj)® 


[(Wy V-) 


(Vj®Wj)® 


(Wj®Wj)} 




= Vy©Wy. 
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Thus W, consists of three pieces, with the following orthonormal bases: 

{^,(%W ^ k lM) 6 Z 2 } o.n.b. in Wj ® V jt 
{<pj, kl (x)%k 2 (y), (h,k 2 ) e Z 2 } o.n.b. in V ; - ® Wy, 
{tpj, kl (x)ip jik2 (y), (k lr k 2 ) € Z 2 } o.n.b. in Wj ® Wy. 

This leads us to define three wavelets 

ft Y(x,y) = *(x)?(y), 
D Y(x,y) = tp{x)tfy), 
d Y(x,y) = tp(x)xp(y). 

Then, { AT Tyk, k = (fci,^) G Z 2 , A = h,v,d} is an orthonormal basis for Wy and { AT Tyk, / G 
Z, k G Z 2 , A = h,v,d} is an orthonormal basis for 0y G ^ Wy = L 2 (IR 2 ). Here, for j G Z, k = 
(itl/fe) G Z 2 and for F G L 2 (R 2 ), the function Fy k is defined as 

F jM (x,y)=2>F(2ix-k 1 ,2>y-k 2 ). 

Now we can proceed and lift the MRA to the sphere. To every function F G L 2 (IR 2 ), one may 
associate the function F s G L 2 (S 2 ) as F s = v • (F o p). In particular, 

Ff k = v - (Fy /k o p) for ; G Z, k G Z 2 , (23) 

and similarly for the spherical functions <& s - k and A Y^ k , where < J ) y / k/ A ^y,k/ h = h,v, d, are the 
planar 2-D scaling functions and wavelets, respectively For j G Z, we define Vy as Vy := 
{v • (F o p), F G Vy}. Then we have: 

(1) Vy C Vy+i for j G Z, and each Vy is a closed subspace of L 2 (S 2 ); 

(2) riyez Vy = {0} and Uy G z Vy is dense in L 2 (S 2 ); 

(3) {<E>q k , k G Z 2 } is an orthonormal basis for Vr> 

A sequence {Vj)j eZ of subspaces of L 2 (S 2 ) satisfying (1), (2), (3) constitutes a MRA of L 2 (S 2 ). 
Define now the wavelet spaces Wy by Vy+i = Vy Wj. Then { A Y s /k , k G Z 2 , A = h,v,d} is 
an orthonormal basis for Wj and { Al Ty k , j G Z, k G Z 2 , A = h,v,d} is an orthonormal basis 

for ( ©y G z Wy J = L 2 (S 2 ). This the orthonormal wavelet basis on S 2 . 

Thus, an orthonormal 2-D wavelet basis yields an orthonormal spherical wavelet basis. 
In addition, if <J> has compact support in IR 2 , then <J>^ k has local support on S 2 (and 
diam supp <& s - k — >■ as ; — >• 00), and similarly for the respective wavelets. Smooth 2-D 
wavelets yield smooth spherical wavelets. In particular, Daubechies wavelets yield locally 
supported and orthonormal wavelets on S 2 . Thus the same tools as in the planar 2-D case can 
be used for the decomposition and reconstruction matrices (so that existing toolboxes may be 
used). 
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Fig. 1. (a) The graph of the function f(6, cp) defined in (24); (b) Its analysis with the spherical 
wavelet associated to the Daubechies wavelet db3, the familiar 6-coefficient filter. 



3.4 An example: Singularity detection 

As an application of our construction, we analyse the following zonal function on S 2 : 



m<p) 



i, 

(l + 3cos 2 ( 



-1/2 



(24) 



The function / and its gradient are continuous, but the second partial derivative with respect 
to has a discontinuity on the equator 6 = j. The function / is shown in Figure 1 (a). 
Detecting properly such a discontinuity requires a wavelet with three vanishing moments 
at least, so that, as far as we know, none of the existing constructions of discrete spherical 
wavelets could detect this discontinuity. 

Instead, we consider the discretized spherical CWT with the spherical wavelet Y^ associated 
to the planar wavelet 



Y„ 2 (x,j/) = A 2 [ e -^ 2+y2) ] 



(25) 



This wavelet has four vanishing moments (again a planar wavelet with less than three vanish- 
ing moments could not detect this discontinuity). The analysis is presented in Figure 2. Panels 
(a), (b), (c) and (d) present the spherical CWT at smaller and smaller scales, a = 0.08, 0.04, 0.02 
and 0.0165, respectively. From Panels (a)-(c), it appears that the discontinuity along the equa- 
tor is detected properly, and the precision increases as the scale decreases. However, there is 
a limit: when the scale a is taken below a = 0.018, the singularity is no more detected prop- 
erly, and the transform is nonzero on the upper hemisphere, whereas the signal is constant 
there. This is visible on Panel (d), which shows the transform at scale a = 0.01655. In fact, the 
wavelet becomes too narrow and " falls in between" the discretization points, ripples appear 
in the Southern hemisphere. This effect is described in detail in (Antoine et al., 2002). 
On the contrary, the well-known Daubechies wavelet db3 lifted on the sphere by (23) does the 
job better than the wavelet Y^ mentioned above, as one can see in Figure 1, Panel (b). The 
computational load is smaller and the precision is much better, in the sense that the width of 
the detected singular curve is narrower. 
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Fig. 2. Analysis of the function f(6, cp) by the discretized CWT method with the wavelet xp s H , 
at scales: (a) a = 0.08 (b) a = 0.04 (c) a = 0.02 (d) a = 0.0165. The sampling grid is 256x256. 2 



The same tests were performed for the function f K /j, obtained from / by performing a rotation 
around the axis Ox with an angle oin/7. The results are presented in Figure 3. Panel (a) shows 
the analysis of the function f K / 7 with the discretized CWT method, using the wavelet ip s H , at 
scale a = 0.0165. Panel (b) gives the analysis with the Daubechies wavelet db3 lifted onto 
the sphere. No appreciable distortion is seen, the detection is good all along the discontinuity 
circle, and again the precision is better with the lifted Daubechies wavelet. Notice that the 
computation leading to the figure of Panel (a) was made with a grid finer than that used in 
Figure 2, so that the detection breaks down at a smaller scale (here below a = 0.01). 
Of course, this example is still academic, but it is significant. More work is needed, in partic- 
ular, for estimating the degree of distortion around the pole and applying the method to real 
life signals. 



4. Generalizations 

As we have seen up to now in the case of the two-sphere, the main ingredients needed for 
construction of a wavelet transform on a manifold are harmonic analysis and a proper notion 
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Fig. 3. (a) Analysis of the function fnpiQ, q>) by the discretized CWT method with the wavelet 
ip s H , at scale a = 0.0165 (the sampling grid here is 512x512); (b) Analysis of the function 
/ n /7(0, cp), with the spherical wavelet associated to db3. 



of dilation on the manifold. Suitable notions of dilation may be obtained by a group-theoretical 
approach or by lifting from a fixed plane by some inverse projection. 

These generalizations do not have a purely academic interest. Indeed, some data live on man- 
ifolds more complicated than the sphere, such as a two-sheeted hyperboloid or a paraboloid. In 
optics also, data on such manifolds are essential for the treatment of omnidirectional images, 
which have numerous applications in navigation, surveillance, visualization, or robotic vi- 
sion, for instance. In the catadioptric image processing, a sensor overlooks a mirror, whose 
shape may be spherical, hyperbolic or parabolic. However, instead of projecting the data from 
that mirror onto a plane, one can process them directly on the mirror, which then suggests to 
use wavelets on such manifolds (Bogdanova, Bresson, Thiran & Vandergheynst, 2007). 

4.1 The two-sheeted hyperboloid H 2 

The upper sheet H 2 + = {£ = (Ci^k) e IR 3 , £? + Q - g = -1, £3 > 0} of the two- 
sheeted hyperboloid may be treated exactly as the sphere, replacing SO(3) by the isometry 
group SO (2, 1). For dilations, however, a choice has to be made, since there are many pos- 
sibilities, each type being defined by some projection. Details may be found in (Bogdanova, 
2005; Bogdanova, Vandergheynst & Gazeau, 2007). Given an (admissible) hyperbolic wavelet 
\p, the hyperbolic CWT of / G L 2 (H^_) with respect to tp is 



w*f(g,«)~(1>gAf) = J I 



H\ 



*a(g-%)f(0M0,g 6 SO (2,1), a> 0, 



(26) 



a formula manifestly analogous to its spherical counterpart (12). As in the spherical case, 
tpa(0 — k(a,£)ip(di/ a £), with d a an appropriate dilation, A(fl,£) is the corresponding Radon- 
Nikodym derivative, and \i is the SO (2, 1) -invariant measure on H 2 . 

The key for developing the CWT is the possibility of performing harmonic analysis on IH 2 ^ , 
including a convolution theorem, thanks to the so-called Fourier-Helgason transform. As a 
consequence, the usual properties hold true, for instance, an exact reconstruction formula. 
However, no result is known concerning frames that would be obtained by discretization. 
On the other hand, it is possible to construct wavelet orthonormal bases on IH 2 ^ by lifting them 
from the equatorial plane £3 = by inverse orthographic (i.e., vertical) projection. In this case, 
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no point has to be avoided, since only one pole is present, but distortions will occur again if 
one goes sufficiently far away from the tip (pole). 

4.2 The paraboloid and other manifolds 

Among the three shapes for a catadioptric mirror, the parabolic one is the most common (think 
of the headlights of a car). And this case brings us back to the topic of Sections 2.2 and 3.3. 
Indeed it has been shown by Geyer & Daniilidis (2001) that the reconstruction of the ortho- 
graphic projection from a parabolic mirror can be computed as the inverse stereographic pro- 
jection from the image plane onto the unit sphere. Thus wavelet frames and wavelet orthogo- 
nal bases may be obtained from the corresponding spherical constructions. Alternatively, one 
may lift planar orthogonal wavelet bases onto the paraboloid directly by inverse orthographic 
projection, as for the hyperboloid, with the same danger of distortions far away. 
For a more general manifold, a local CWT may be designed, using a covering of the manifold 
by local patches (charts, in the language of differential geometry) and the projection along the 
normal at the center of each patch (Antoine et al., 2009) (this is also the idea behind needlets 
(Narcowich et al., 2006b)). One would then get orthogonal wavelet bases in each patch, but 
there remains the problem of connection of one patch with the next one, using transition func- 
tions (the concatenation of all the local bases may also be considered as a dictionary). No- 
tice the same problem of combining local orthogonal wavelet bases has been encountered, 
and solved, in the wavelet construction based on radial projection from a convex polyhedron 
(Ro§ca, 2005), described briefly in Section 3.2(3). 

A final example of orthogonal wavelet basis is that of the wavelet transform on graphs (An- 
toine et al., 2009). A graph is a good model for pairwise relations between objects of a certain 
collection, such as the nodes of a sensor network or points sampled out of a surface or mani- 
fold. Thus a wavelet transform on a graph could be a welcome addition. 
A graph is defined as a collection V of vertices or nodes and a collection of edges that connect 
pairs of vertices. In the present context, one considers finite graphs only, with d nodes. Thus 
the signals of interest are functions / : V — » R, which can be identified with ^-dimensional 
real vectors / £ K d . In order to design a wavelet transform on such a graph, one considers the 
so-called Laplacian matrix, a positive semi-definite d x d matrix. Its eigenvectors form an or- 
thonormal system that can be used to decompose any signal. Next one defines a dilation by di- 
lating the "Fourier" coefficients — once again the needlet idea. The resulting functions are the 
wavelets on the graph and they form an orthogonal basis (everything is finite-dimensional). 
We refer to (Antoine et al., 2009) for further details of the construction. 

5. Outcome 

We have surveyed a number of techniques for generating orthogonal wavelet bases or wavelet 
frames on the two-sphere S 2 , plus some generalizations. Two approaches have been privi- 
leged, both of them based on some notion of inverse projection, namely, (1) the construction 
of a CWT on S 2 by inverse stereographic projection from a tangent plane, which leads to 
nontight frames upon discretization; and (2) the construction of orthogonal wavelet bases by 
lifting in the same way a planar orthogonal basis. Of course, many other methods are avail- 
able in the literature, especially in the discrete case, and we have mentioned some of them. 
Clearly many open questions remain, but we want to emphasize that progress in this field is 
likely to be motivated by physical applications, in particular, astrophysics and optics. 
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1. Introduction 

Multiple antenna communications technologies offer significant advantages over single 
antenna systems. These advantages include extended range, improved reliability in fading 
environments and higher data throughputs. If multiple antennas are provided only at the 
transmitting end of a link then the system is referred to as multiple input single output 
(MISO). If multiple antennas are provided only at the receiving end of a link then the system 
is referred to as single input multiple output (SIMO). If multiple antennas are provided at 
both ends of a link then the system is referred to as multiple input multiple output (MIMO). 

Multiple antenna systems can be divided into two classes depending on the signal 
processing employed. These are: (i) smart antennas and (ii) spatial multiplexors. 

Smart antennas provide increased signal-to-noise-and-interference ratio (SNIR) via diversity 
gain, array gain and/ or interference suppression. Each transmit antenna radiates, to within 
a simple gain and delay difference, the same signal. Similarly, each receive antenna 
contributes its signal to a gain and delay weighted sum. By setting transmit and receive 
gains and delays appropriately, improved SNIR is achieved which may be used to realise 
greater spectral efficiency (and, therefore, greater channel capacity), greater range and/ or 
decreased latency (due to a reduced requirement for channel coding). 

Spatial multiplexors can provide increased channel capacity directly. Each transmit antenna 
radiates an independent signal sub-stream. With N transmit and N receive antennas, for 
example, an N-fold increase in data-rate is possible (in principle) over that achievable with a 
single input single output (SISO) antenna system (without any increase in total transmitted 
power). 

In this chapter a theoretical framework for describing MIMO channels is presented followed 
by a brief outline of MIMO channel modelling principles [P. Aimer et. al., 2007]. The rest of 
the chapter describes a selection of widely adopted MIMO channel models, their capabilities 
and limitations, in the context of specific standards. 
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2. MIMO channel framework 

Wireless channels are linear. They may, therefore, be represented by a linear filter and 
described by their impulse response. If the transmit and receive antennas, and the scattering 
objects in the environment, are static then the channel will be time-invariant and the 
impulse response, h(r), will be a function of delay, r, only. If the transmit antenna, receive 
antenna or scattering objects move then the channel will be time-variant and the impulse 
response, h(t, r), will be a function of time, t. 

From a communication systems point of view h(t, r) is an entirely adequate channel 
description, i.e. it represents sufficient information to predict the (noiseless) received signal 
from the transmitted signal. It has the limitation, however, of obscuring the underlying 
propagation physics. If h(t, r) is to be predicted from full or partial knowledge of the 
physical environment, then models linking it to the most important aspect of the 
environment, i.e. the spatial distribution of scattering objects, is required. The double- 
directional impulse response is a channel description that makes explicit this connection 
between the systems-level impulse response and propagation physics. 

2.1 Double-directional impulse response 

The impulse response of the wireless transmission channel describes the cascaded effect of 
transmit antenna, propagation channel and receive antenna. Changing an antenna, 
therefore, may change the impulse response, even though the propagation channel (i.e. the 
physical arrangement of scatterers) may remain the same. The effects on h(t, r) of the 
antennas and propagation environment can be decoupled using a description of the channel 
called the double-directional impulse response [M. Steinbauer et. al., 2001], i.e.: 

A ('' r ) = J \sT^T)P( t ^^R^T)S R i^R)d^ T d^ R 

AnAn n\ 

where the integrand is the component of impulse response represented by power leaving 
the transmit antenna with (vector) direction 97 (or, more strictly speaking, within the 
element of solid angle, d<pT, centred on 97) and arriving at the receive antenna with (vector) 
direction 9^. gR(<pR) and gr(9r) are the complex (field-strength or voltage) gains of the 
receive and transmit antennas, respectively, in the directions 9^ and 97. (The magnitude of 
the voltage gain is the square root of the conventional antenna (power) gain and is 
proportional to an antenna's effective length - the square root of its effective area - via the 
antenna reciprocity formula.) For isotropic antennas gR(9#) = gr(9r) = 1. p(t, r, 9r, 9r) is the 
component of impulse response assuming isotropic antennas. It incorporates all propagation 
related losses including free-space path loss, absorption and scattering loss. The integral in 
Eq. (1) sums over all possible directions-of-departure (DODs) at the transmitter and all 
possible directions-of-arrival (DOAs) at the receiver. 

If propagation is via K discrete paths then Eq. (1) can be written as a summation over K 
multipath components (MPCs), i.e.: 
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h (t^) = ^g T ((f T )p(t,T,(f R ,(f T )g T ((f T ) 

(2) 

Recognising that the channel characteristics are a function of transmit and receive antenna 
locations, denoted by the position vectors rj and tr, respectively then the double-directional 
impulse response may be more fully expressed using p(rr, tr, t, r, (p#, 91), i.e.: 

K 

h(r T ,r R ,t,T) = ^g T ((p T )p(r T ,r R ,t,T,(p R ,(p T )g T ((p T ) 

(3) 

The description above relates to singly-polarised antennas. A more complete channel 
description can be devised by introducing a matrix of impulse responses defining the 
coupling between, for example, the vertically- and horizontally-polarised ports of a dual- 
polarised transmit antenna and the vertically- and horizontally-polarised ports of a dual- 
polarised receive antenna, i.e.: 



Ht,r) = 






(4) 



The leading diagonal (co-polar) elements of Eq. (4) describe coupling between the vertically 
polarised port of the transmit antenna and the vertically polarised port of the receive 
antenna {h vv ), and the horizontally polarised port of the transmit antenna and the 
horizontally polarised port of the receive antenna (huh). The off diagonal (cross-polar) 
elements describe coupling between the horizontally polarised port of the transmit antenna 
and the vertically polarised port of the receive antenna Qi v y), and the vertically polarised 
port of the transmit antenna and the horizontally polarised port of the receive antenna (hh v ). 
In principle, the ports could have any pair of orthogonal polarisations. In practice, however, 
they are almost always perpendicular linear polarisations (typically vertical and horizontal 
for terrestrial links) or counter-rotating (right-handed and left-handed) circular 
polarisations. 

Eq. (4) is a systems description. To express the cross-polarising effects of the antenna and 
propagation medium separately a cascade of three polarisation matrices (one each for 
transmit antenna, medium and receive antenna) is required for each propagation path. The 
system matrix for a discrete set of propagation paths is then given by: 
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KAt,T) h vh (t,T) 



=z 



P^{t,T,^ R ,^ T ) p vh (t,T,ip R ,<p T ) 

&7>v,(<Pr) £?>/,, (<Pr) 



(5) 



The use of vertical and horizontal basis polarisations (or, more generally, any pair of 
perpendicular linear polarisations) in Eq. (5) is problematic. A simple cartesian definition of 
a pair of basis polarisations (definition 1, Fig. 1) is adequate only for the single propagation 
path described by cpR = cpr = 0. For propagation paths with non-zero DODs and DOAs some 
other pair of basis polarisations must be adopted [A. C. Ludwig., 1973]. The polarisations of 
a pair of dipoles (one electric, one magnetic) can be used - definition 2, Fig. 1 - as can the 
polarisations of a pair of perpendicular Huy gens' sources - definition 3, Fig. 1. (A Huy gens' 
source is the elementary radiating source of an electromagnetic wave referred to in Huygens 
' Principle which states that each point on a propagating wave-front acts as a secondary 
source of radiation.) 



Definition 1 



Definition 2 



Definition 3 




L^« 




Uy 





i,Y 





Fig. 1. Lud wig's three definitions of orthogonal basis polarisations (After [A. C. Ludwig, 
1973]. (©1973 IEEE)) 

Definitions 2 and 3 allow a pair of perpendicular polarisations to be defined for directions 
other than (p# = 97 = whilst preserving the requirement for transverse electromagnetic 
fields. They still suffer from limitations, however, since definition 2 does not define 
polarisation in the ±Y direction and definition 3 does not define polarisation in the -Z 
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direction. (This is reflected physically by the fact that a dipole does not radiate along its axis 
- it has a toroidal radiation pattern - and a Huygens source does not radiate in the backward 
propagating direction - its radiation pattern is a cardioid of revolution.) 

For radio links with columnated-beam antennas, and DODs and DOAs closely clustered 
about transmit and receive antenna boresights, the most appropriate choice of basis 
polarisations is probably definition 3. (As the spread DODs and DOAs about boresight 
tends to zero then all three definitions converge.) For links with omnidirectional antennas, 
and DODs and/ or DOAs widely spread in azimuth (but not too widely spread in elevation), 
then definition 2 is probably more appropriate. 

2.2 The MIMO channel impulse response 

Fig. 2 shows a schematic diagram of a MIMO channel. 



Transmit Antennas 




Receive Antennas 



Channel Model 



Fig. 2. MIMO channel 



The MIMO channel must be described for all transmit and receive antenna pairs. For M 
transmit antennas and N receive antennas the MIMO transmission channel can be 
represented by an N x M channel matrix, i.e.: 
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Ht,T): 



hn(t,T) h l2 (t,T) 
h 21 (t,T) h 22 (t,T) 



h Nl (t,T) h N2 (t,r) 






r(t,T) 



(6) 



where h nm (t, r) represents the time-variant impulse response between the input of the m th 
transmit antenna and the output of the n th receive antenna. Each impulse response is the 
cascaded effect of transmit antenna, the propagation medium and a receive antenna. This is 
therefore a system-level representation. If polarisation diversity is employed then each 
element of the matrix must be replaced by the polarisation matrix of Eq. (4). This is 
equivalent to doubling the number of antennas at each end of the link; a dual-polarised 
antenna being treated, effectively, as two singly-polarised antennas. Any time variation, due 
to shadowing and/ or multipath fading, arises from antenna motion or the motion of 
environmental scatterers. The spatial, polarisation and temporal correlations between the 
signals at the terminals of different receiving antennas are reflected in the correlation 
properties of the matrix elements. 

MIMO channel models may be physical or analytical. Physical models are based either on a 
physical theory (often geometrical optics) or on physical measurements. They are site- 
specific or specific to an environment type (e.g. urban suburban, rural) and are particularly 
useful in network planning. Analytical models are based on mathematical assumptions 
about channel behaviour. They are generally site-independent and are mostly used for 
system design, comparison and testing. 

Physical models can be subdivided into deterministic and stochastic variants. Deterministic 
models are environment specific and are derived from the underlying physical radio 
propagation processes, e.g. reflection, diffraction, shadowing and wave-guiding. The most 
important example is ray tracing. Stochastic (physical) models are more generic than 
deterministic models. They are based on the fact that whilst, in the absence of a detailed 
environment database, the precise physical propagation parameters (e.g. DOD, DOA, 
number of paths, path delay, path power) are unpredictable, they nevertheless have well- 
defined statistical behaviours. Probability models can therefore be constructed for these 
propagation parameters. They are generally more computationally efficient than 
deterministic models. The Spatial Channel Model (SCM) and the Spatial Channel Model - 
Extended (SCME) described in sections 4.1.1 and 4.1.2 are stochastic physical models. 



Analytical channel models derive the MIMO channel matrix without any consideration of 
propagation parameters. Examples include Independent Identically Distributed (i.i.d.), 
Weichselberger and Kronecker models. The WiMAX and IEEE 802.1 In channel models 
described in sections 4.2 and 4.3 are Kronecker models. Since MIMO channel matrices are 
easily generated using analytical models, and since the statistics of these matrices are both 
unvarying (repeatable) and environment independent, they are popular for the 
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development, verification and optimisation of system hardware and software (especially 
signal processing algorithms). 

3. Link-level and system-level simulations 

Most standardised MIMO channel models provide link-level and system-level simulation. 
The former refers to a single point-to-point link (but including, of course, multiple transmit 
and receive antennas). The latter refers to multiple communication links, potentially 
including multiple base-stations (BSs). In the case of an SCM channel, for example, a 
calibration process is undertaken at the link-level prior to a system-level simulation. 

The SCM and SCME system-level model defines a 'drop' concept where a mobile is placed 
(dropped) in a sequence of different network locations. The locations may be random or pre- 
defined by the user. Each drop represents a snapshot of the fading channel. The statistics of 
the channel parameters within a single drop are assumed to be stationary and, for the 
duration of the drop all large scale parameters (DOA, DOD, mobile station velocity etc.) are 
assumed to be constant. The drop concept is illustrated in Fig. 3. 




Fig. 3. Quasi-stationary drop periods in a channel simulation 



A variation of the drop-based simulation allows the large-scale channel parameters to 
evolve (change gradually and continuously) at each simulation step inside a drop. This 
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approach is more realistic than the stationary assumption but represents an increase in 
complexity and results in increased simulation time. 

If motion of the mobile station (MS) and scatterers is assumed to be negligible during the 
course of a packet transmission the channel is said to be quasi-static. Such a channel has a 
decorrelation time, T c , which is greater than the packet duration. If T c is defined as the time 
shift resulting in a correlation coefficient of 0.5 then it may be related (approximately) to the 
maximum Doppler shift, f m (speed/ wavelength) [T. S. Rappaport., 2002], by: 



T «- 



l67 fm 



(7) 



For a carrier frequency of 2 GHz and a terminal speed of 3 km/h the maximum Doppler 
shift would be 5.6 Hz and the channel decorrelation time would be 32 ms. The assumption 
that the channel is quasi-static could then be justified providing the transmitted packet 
duration is less than this. 



4. Channel model standards 

Specific standardised channel models have been developed for the testing and optimisation 
of particular wireless standards in a manner that is repeatable and widely agreed. These 
wireless channel models are important for the development of new wireless devices. 
Examples of standardised channel models include COST 207, SCM and SCME. These are 
wideband power delay profile (PDP) models used in the development of GSM, WCDMA 
and LTE systems, respectively. Standardised channel models provide a framework to test 
algorithms and investigate design trade-offs thereby informing key design decisions relating 
to modulation, coding and multiple-accessing etc. 

A selection of standardised channel models for mobile, broadband wireless access (BWA) 
and wireless local area network (WLAN) applications is given below: 



Channel Model 


Origin 


Application 








SCM 


3GPP/3GPP2 


3G Outdoor 


SCME 


WINNER 


3G Outdoor 


WIN II 


WINNER II 


3G Outdoor 


SUI 


Stanford University 


Fixed BWA 


WiMAX- ITU-TDL 


WiMAX form 


Fixed/ Mobile BWA 


IEEE 802.11n 


IEEE 802.1 In TGn (High throughput task group) 


Indoor Channel 



4.1 WINNER 

The European WINNER (wireless world initiative new radio) project began in 2004 with the 
aim to develop a new radio concept for beyond third generation (B3G) wireless systems. 
Work Package 5 (WP5) of the WINNER projects focused on multi-dimensional channel 
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modelling for carrier frequencies between 2 and 6 GHz and bandwidths up to 100 MHz. In 
total six organisations were formally involved in WP5 (Elektrobit, Helsinki University of 
Technology, Nokia, Royal Institute of Technology (KTH), the Swiss Federal Institute of 
Technology (ETH) and the Technical University of Ilmenau. 

At the start of the project there was no widely accepted channel model suitable for WINNER 
system modelling. Two existing channel models, 3GPP/3GPP2 Spatial Channel Model 
(SCM), were selected as starting points for outdoor simulation and one existing channel 
model (IEEE 802.11 TGn Model [3GPP TR25.996 V6.1.0, 2003-09]) was selected as a starting 
point for indoor simulation. The SCM had insufficient bandwidth and too few scenarios. In 
2005 the first extension of SCM, SCM Extended (SCME), was therefore proposed. Despite 
the modifications to SCM, SCME was deemed inadequate for the simulation of B3G systems. 

At the end of 2005 the WINNER channel model - Phase 1 (WIM1) was described in the 
deliverable D5.4 [WINNER, 2005]. WIM1 has a unified structure for indoor and outdoor 
environments and is based on double-directional measurement campaigns carried out in the 
5 GHz ISM2 band with bandwidths of up to 120 MHz. It covers six different propagation 
scenarios, i.e.(i) indoor small office, (ii) indoor hall, (iii) urban microcell, (iv) urban 
macrocell, (v) suburban macrocell, and (vi) rural . Both line-of-sight (LOS) and non-line-of- 
sight (NLOS) propagation conditions are catered for [H. El-Sallabi et. al., 2006]. 

In September 2007, the WINNER channel model - Phase II (WIM2) was described [WINNER 
II interim, 2006]. This model, which evolved from WIM1 and the WINNER II interim 
channel models, extended the propagation scenarios to: (i) indoor office, (ii) large indoor 
hall, (iii) indoor-to-outdoor, (iv) urban microcell, (v) bad urban microcell, (vi) outdoor-to- 
indoor, (vii) stationary feeder, (viii) suburban macrocell, (ix) urban macrocell, (x) rural 
macrocell, and (xi) rural moving networks. In the course of the WINNER project channel 
models were implemented in MATLAB and made available through the official web site 
[WINNER II, 2007]. 

4.1.1 Spatial Channel Model (SCM) 

SCM was developed by 3GPP/3GPP2 (third generation partnership project) for outdoor 
environments at a carrier frequency of 2 GHz. It was designed to test 5 MHz CDMA 
channels [P.Almer et. al., 2007] and consists of two parts: (i) a link level simulation model 
and (ii) a system-level simulation model. The SCM system level model is currently used as a 
de-facto standard for LTE, WCDMA, UMTS and WiMAX system level evaluation and 
performance verification. 

4.1.1.1 Link-level model 

The link-level model which might also be referred to as a reference model is a single link 
channel model. It provides a well-defined, convenient, interface for different equipment 
manufacturers to compare their proprietary implementations of the same signal processing 
algorithms. Link-level simulations alone are not recommended for performance testing of 
different algorithm because they reflect only a single snapshot of the (dynamic) channel. 
Link-level simulations do not, therefore, allow conclusions to be made about the general 
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behaviour of a system and if such conclusions are required system-level simulations must be 
performed. 

The link-level SCM can be implemented as either a stochastic physical or analytical model. 
In the former the wideband characteristics of the channel are modelled as a tapped delay 
line (TDL). Each tap is independently faded and is characterised by an azimuth DOD/DOA 
angular spectrum described by a uniform distribution (for MSs) or a Laplacian distribution 
(for BSs). The mean direction and angular spread at BS and MS are fixed (and thus represent 
stationary channel conditions). The Doppler spectrum is calculated based on the MS velocity 
(speed and direction relative to the line connecting MS and BS). The model also defines the 
number and configuration of antennas at MS and BS. Given all these parameters the 
physical model can be transformed to an analytical model [P. Aimer et. al., 2007]. 

4.1.1.2 System-level model 

The system-level model is a multi-link physical model intended for performance evaluation 
in which each link represents a cell or a sector within a cell. Fig. 4 illustrates a system-level 
simulation in which an MS receives interference from adjacent sectors of adjacent cells. 




ig. 4. SCM system level simulation. 
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Each link comprises an MS and BS MIMO antenna array. Propagation is via multipaths and 
sub-paths. The excess delays of sub-paths are closely clustered around the delay of their 
(parent) multipath. This is assumed to originate from an environment with closely spaced 
clusters of scatterers. Fig. 5 illustrates the clustered scatterers and resulting multipaths and 
sub-paths. 

The SCM distinguishes between three different environments, i.e. (i) urban macrocell, (ii) 
suburban macrocell and (iii) urban microcell. The modelling and simulation methodology 
are identical for all three environments but the parameters (e.g. azimuth spread, delay- 
spread, shadow fading and path loss) are different [P. Aimer et. al., 2007]. 




Array 



BS Array 



Delay Profile [ns] 



Fig. 5. Clustered scatterers, multipaths and sub-paths in an SCM simulation. 



4.1.1.3 Software implementation of SCM 

The modelling approach can be divided in to three parts: (i) antenna correlation (ii) spatial 
correlation and (iii) polarisation correlation. This is illustrated schematically in Fig. 6 which 
represents the impulse response of the n th multipath component. 
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Fig. 6. Tap n complex gain for SCM 



Where H U/S , n (t) is the complex gain of the n th tap (corresponding to the n th MPC) between the 
s th element of a linear BS array and the u th element a linear MS array. x v Bs(0n,m,DOD) and 
x h Bs(6n,m,DOD) are, respectively, the BS antenna complex field patters for vertically and 
horizontally polarised fields. Matrix a„ /W represents cross-polarised coupling between 
vertically and horizontally polarised components. x v Ms{Qn,m,DOA) and x h Ms{Qn,m,DOA) are, 
respectively, the MS antenna complex field patterns for vertically and horizontally polarised 
fields. (Note that the DOD and DOA are each defined by a single angle. This reflects the 
assumption that there is spreading in azimuth only. Whilst energy is unlikely to be 
completely constrained to a single azimuthal (horizontal) plane in reality the spread of 
energy in elevation (out of the horizontal plane) is likely to be relatively modest. The spatial 
correlation is a function of separation between BS and MS antenna array elements and their 
respective DODs and DOAs. A random phase offset cp n , m (uniformly distributed between 
and 360°) is added to ensure a random starting point for fast fading. Temporal correlation is 
a function of the magnitude and direction of MS motion. Fig. 7 shows a pseudo-code flow 
diagram for the model. The simulation steps implied by Fig. 7 are: 
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Legend 

L = Number of MIMO links 
N = Number of Multipaths 
M = Number of Subparts 
T = Number of Fading Samples 




■f True 



Initialisation 

and BS Distance Depends on Cell Layout 

MS Antenna Array Orientation Uniform between ff to 360° 

MS Direction U n iforrn between CP to 3S0° 

BS and M$ Antenna Gain Pattens Path -ad over 360* in Azimuth 



IE 



Detemnine Channel Parameters (based on user input) 


Path Delay 


Uniformly Distributed 


Average Power In each Tap 


Gaussian Distributed 


Direction of Departure [for each multipath) 


Gaussian Dislributed 


Clustering (of each multipath) at the BS 


Detemnine Power. Phase and Offset 




for each Subpalh in a Cluster. 


Direction of Arrival {for eachi multipath) 


Gaussian Dislributed 


Clustering {of each multipath) at the MS 


Determine Power, Phase and Offset 




for each Su bpatfi in a Cluster. 


Pairing of Multipath and Subpaths 


Random 


Palbkjss and Shadowing 


Deterministic Models 


Random Phase for each Link 


Uniform Distributed 



Antenna Field Pattern 
Interpolation 



Interpolate MS and BS Antenna 

Gain Patterns based on user 

Specified Interpolation Techniques 

recommended (Cubic, Linear) 




For eachi link generate and store the final phase 
associated with each multipaths and subpaths 



Scale Fast Fading Samples by Path Loss and Shadow Fading 



c 

Fig. 7. Pseudo-code for SCM channel model 
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1. For a given drop the multiple MSs are placed at random locations within a 
sector/ cell. The MS (linear array) antenna orientation and the MS direction of 
motion are selected at random. 

2. Path-loss is calculated based on the separation between MS and BS. The path-loss 
model used is the COST 231 (Hata) model for macrocells and the COST 231 
(Walfish-Ikegami) model for microcells. The number of multipath components is 
fixed in all three cases at six and their delay and average power are chosen 
randomly using appropriate probability distributions. 

3. Angular dispersion at the MS and BS is incorporated by assuming each multipath 
component comprises a cluster of 20 sub-paths having the same delay but 
(randomly modelled) different DOD and DOA [3GPP TR.996 V6.1.0 (2003 09)]. 
The overall mean DOD and DOA is determined by the relative locations of BS 
and MS and the orientation of their antenna arrays. The mean DOD or DOA for 
each tap is chosen at random from a Gaussian distribution that is centred on the 
overall mean. (DOA and DOD variances are model parameters.) . The sub-paths 
have deterministic amplitude and random phase. Their sum is therefore subject 
to Rayleigh or Ricean fading. 

4. Temporal fading may be generated either by using a sum of sinusoids or by 
applying white Gaussian noise to a Doppler filter. 

5. Temporal variation of the impulse response is determined by the speed and 
direction of the MS. The different Doppler shift for each sub-path leads to a 
different phase for each sub-path. The phase for each sub-path at the end of each 
call within a single simulation drop is stored in order to ensure continuity, Fig. 8. 



1 fil call to Fading 
Samples 


Mm 


2 nd call to Fading 
Samples 


■ 


3 rd call to Fading 
Samples 


■ 




Number of Samples 

Fig. 8. Fast fading samples for one tap corresponding to successive calls of the same 
channel drop. 



For system level simulations (which consider multiple MSs and BSs in different 
cells/ sectors) a sequence of drops is executed. All parameters are independent of 
those in prior or succeeding drops. The drop period is assumed to be sufficiently 
short for large-scale channel parameters (e.g. angle spread, mean DOD/ DOA, 
delay-spread and shadowing) to be constant during the drop. The MS position is 
determined at random during the start of each drop. 



MIMO Channel Modelling 91^ 

4.1.1.4 Model features 

1. Each drop reflects a snapshot of the fading channel. 

2. The COST 231 Urban Hata (macrocell) and COST 231 Walfish-Ikegami (microcell) 
path-loss models are adjusted for a frequency of 1.9 GHz. The applicability to a 
frequency other than 1.9 GHz is not analysed [M. Narandzic, 2007]. 

3. The model provides a bandwidth of 5 MHz which is not adequate for the most 
recent wireless standards (e.g. LTE which requires bandwidth of 20 MHz). 

4. During a drop the channel undergoes fast fading due to MS movement. Delays, 
DOD and DOA, however, are kept constant. Any two consecutive drops are 
independent and are based on randomly located clusters. This makes the channel 
model discontinuous across drops. 

5. Six clusters of scatterers are considered. Each cluster corresponds to a resolvable 
MPC (referred to as a multipath). Within a resolvable path (cluster), there are 20 
irresolvable sub-paths. Each of the multipaths is modelled as a Dirac (delta) 
function of delay. Each multipath is subject to angular dispersion across its 20 
sub-paths. The summing of the sub-path carriers results in Rayleigh fading of 
each multipath [D.S. Baum., 2005]. 

6. The correlation of the standard deviation (in dB) of lognormal fading due to 
shadowing (between the links from a single MS to multiple BSs) is 0.5. The 
correlation is independent of the range of BSs or their relative angular locations 
as seen from the MS. Shadowing and its correlation between multiple MS-BS 
paths are therefore assumed independent of network topology and topography 
[3GPP TR25.996 V6.1.0, 2003-09]. 

7. Elevation spread is not considered. 

8. Antenna radiation pattern, array geometry and array orientation are arbitrary. 
When all propagation and antenna parameters are defined an analytic 
formulation can be extracted from the physical model. Each drop results in a 
different instance of the correlation matrix. 

9. Most model parameters are described by their PDFs. While this provides a rich 
source of variability, it has the disadvantage that simulation time grows 
exponentially with the number of random parameters. 

10. As a consequence of the assumption that each multipath is flat-faded, the model's 
utility for band widths above 5 MHz is questionable. (This motivates the 
extended, SCME, version of SCM.) 

11. Uplink and downlink reciprocity is assumed, i.e. DOD and DOA values are 
identical for both uplink and downlink simulation of the channel. 

12. The uplink and downlink sub-path (random) phases are uncorrelated for 
frequency division duplex (FDD) systems but correlated for TDD systems. . 

4.1.2 Extended Spatial Channel Model (SCME) 

SCME is an extension of SCM. The extension is not associated with the 3GPP working group 
but was developed in WP5 of the WINNER project. SCME extends the channel bandwidth 
of SCM from 5 MHz to 20 MHz. It was adopted as the channel standard for the 
development and testing of the 3GPP Long Term Evolution (LTE) standard. The channel 
bandwidth was subsequently further extended to 100 MHz. (The required bandwidth of 
B3G systems is up to 100 MHz in both 2 GHz and 5 GHz bands.) A limitation of SCM is the 
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drop-based approach with the consequence that there is no short-term variability in the 
channel transfer function. This corresponds to fixed DOAs as seen by a moving MS. SCM 
also has a limited range of scenarios (it does not include, for example, outdoor-to-indoor 
paths) and, in some scenarios, does not incorporate K-factor to support LOS paths. 

SCME uses the intra-cluster delay-spread to effect bandwidth extension. Since backward 
compatibility with SCM was required the number of clusters (i.e. multipaths) is not 
increased from six. Each cluster of 20 sub-paths (which in SCM have identical delay) is 
subdivided into 3 or 4 sub-clusters (for macrocell and microcell scenarios respectively) 
called mid-paths with different delays, Fig. 9. 
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Fig. 9. SCME multipaths, mid-paths and clusters 



(The total number of sub-paths for SCME, however, remains the same as for SCM, i.e. 20.) 
Bandwidth extension is realised by introducing delay (and power) difference between the 
mid-paths The number of delay taps is therefore increased from 6 in SCM to 18 or 24 in 
SCME (depending on the scenario [Spirent Communications, 2008]). The 20 sub-paths are 
split into groups of 10, 6, and 4 (for scenarios with three mid-paths) or groups of 6, 6, 4 and 4 
(for scenarios with four mid-paths). The relative power of each mid-path is scaled by this 
ratio as shown in Fig. 10. 
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Fig. 10. Multipaths, mid-paths and sub-path in SCME. 

Sub-paths within a mid-path have identical delay. Each mid-path has the same azimuth 
spread as in SCM. SCME is a continuous evolution model since it allows drifting of DODs, 
DO As and delays for every multipath at each simulation step within a drop. 

4.1.2.1 Additional features of SCME 

SCME contains the following additional features when compared to SCM [D.S Baum et. aL, 
2005; M. Narandzic, 2007; SCME Project., 2005]: 

1. Addition of intra-cluster delay-spread within a multipath 

The introduction of intra-cluster delay-spread to increase bandwidth. The 20 subpaths of 
SCM are grouped into three (in case of macrocells) or four (in case of microcells) mid-paths 
with different delays. This results in about 10 ns of intra-cluster RMS delay spread. The 
delays, which are fixed, are given in [D.S Baum et aL, 2005]. A mid-path, which represents a 
single resolvable delay (or tap), comprises a collection of sub-paths and therefore has a 
fading distribution close to Rayleigh. 

2. Frequency 

Carrier frequency is extended to 5 GHz. 

3. Path-loss models 

Two path-loss models are used. The long-range model (identical to that defined in the SCM 
2 GHz model) and an alternative, short-range, model. The extension of frequency to 5 GHz 
is applied to both. SCME selects the path-loss model (long-range/ short-range/ 2 GHz/ 5 
GHz) depending on the user input. The 5 GHz path-loss model offsets loss by +8 dB with 
respect to the existing 2 GHz model. 



4. LOS model for all scenarios 
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The LOS option affects path-loss and shadow-fading variance. The choice between LOS and 
NLOS within a drop is based on the probability of LOS versus BS-MS distance. The LOS 
model in SCM defines a path-loss and Ricean K-factor which is applicable to urban microcell 
scenario only. With the alternative path-loss model in SCME, the LOS option (with 
appropriate K-factor model) is defined for all scenarios and is thus available whenever the 
current drop is LOS. 

5. Time-variant shadowing 

Each drop consists of time samples within a channel snapshot. In SCM the fading due to 
shadowing is constant for the duration of a drop. In SCME shadow induced fading (in dB) 
changes within a drop thus modelling time-varying shadowing. The decorrelation distance 
of shadowing is predefined (i.e. 5 m, 50 m and 250 m in urban microcells, urban macrocells, 
and suburban macrocells, respectively). The standard deviation of shadow induced fading 
for all scenarios is 4 dB for LOS and 10 dB for NLOS. 

6. Time-variant DODs, DOAs and delays 

For all sub-paths, DODs and distances are calculated once for every channel snapshot. It is 
assumed, therefore, that the locations of scatterers relative to the BS are fixed for the 
duration of a drop. This results in fixed DODs as seen by the BS (with the exception of any 
LOS DOD which does change). The DOAs as seen from the MS and the sub-path delays 
change during a drop due to the MS movement. 

4.1.2.2 Model features 

1. SCME is a stochastically controlled spatial channel model. The model is based on 
the same design philosophy as SCM i.e. the summation of specular components 
to define the changing impulse response. The model is backward compatible 
with the existing SCM model. 

2. SCME provides a channel bandwidth of up to 100 MHz which is sufficient to 
characterise B3G wireless technologies and networks. 

3. In addition to fast fading (due to multipath propagation), SCME can model the 
evolution of slow fading (due to shadowing), sub-path delay and DOA during 
each drop. This results in, time-varying, spatial correlations between transmit and 
receive antenna elements as shown in Fig. 11. 
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Fig. 11. Tap n complex gain for SCME 



4. 



5. 



6. 



SCME considers six clusters of scatterers. Each cluster corresponds to a resolvable 
path. Within a resolvable path (cluster), there are 20 sub-paths. The 20 sub-paths 
are divided into mid-paths which are then assigned different delays relative to 
the original path. The mid-paths are delay resolvable but sub-paths within a mid- 
path are irresolvable. The mid-path (which is a collection of subpaths) 
corresponds to a single tap. 

SCME includes a LOS K-factor option which is switch selectable for all urban and 
suburban macrocell scenarios. (SCM includes a LOS model for the urban 
microcell scenario only.) 

A simplified tap delay line model referred to as the cluster delay line (CDL) 
model is used for calibration and comparison purposes. In this model, the DODs 
and DOAs are fixed for each path. Also, fixed delays are defined resulting in a 
fixed power delay profile (PDP) [Spirent Communications, 2008]. 



4.1.2.3 Software implementation of SCME 



1. 



2. 



The drop concept in SCM corresponds to a relatively short channel-observation 
period that is significantly separated from adjacent drops in both time and space. The 
channel parameters are therefore constant within the drop and independent between 
drops. Short-term variability of channel parameters within a drop is incorporated by 
introducing drifting of (i) path delays (ii) DOAs and (iii) shadow induced fading. 
The position of scatters is fixed within a drop. As a consequence the scattering 
angles as seen from the BS (DOD) do not change (with the exception of the LOS 
DOD in LOS scenarios). This assumption is common to SCM. The initial values of 
random parameters such as DODs, DOAs, K-i actor, path phases etc. are 
generated in the same way as in SCM. 

The scatter angles as seen from the MS (DOAs) and sub-path delays change (in 
contrast to SCM) during a drop reflecting MS movement. Similarly the LOS 
direction from the BS to MS varies in time. This results in time-varying spatial 
correlation between MS and BS antenna array elements. 
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4. An initial value of distance (djj) between the MS and the last bounce scatter (LBS) 
of the i th sub-path of the j th multipath is required in order to calculate DO A drifts 
as the MS moves. This distance is unknown but can be inferred from a stochastic 
model as proposed in [D. S. Baum et. aL, 2005]. 

5. Time evolution, or drifting, of slow fading (shadowing) is determined by the 
spatial auto-correlation function. 

Fig. 12 shows the pseudo-code flo w diagra m for an implementation of SCME. 

( Start ^ 
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Fig. 12. Pseudo-code flow diagram of SCME channel model. 
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4.1.3 WIM2 

The WIM2 channel model (also referred to as WINNER II [WINNER II Channel Models, 
2006]) is defined for both link-level and system-level simulations. It encompasses a wide 
range of scenarios relevant to local, metropolitan and wide-area systems. WIM2 evolved 
from the WINNER I and WINNER II (interim) channel models. 

WIM2 is a double-directional geometry-based stochastic channel model. It incorporates 
generic multilink models for system-level simulations and clustered delay line (CDL) 
models, with fixed large-scale channel parameters, for calibration and comparison purposes. 

Initially, the WINNER group selected SCM for immediate simulation but later extended this 
to SCME. In spite of the greater bandwidth and higher frequency capability of SCME it was 
still deemed inadequate for advanced WINNER II simulations. The novel features of WIM2 
are its parameterisation, the addition of further outdoor and indoor scenarios and the 
consideration of both azimuth and elevation spreading for indoor environments. (This is in 
contrast to SCM and SCME which restrict spreading to the azimuthal plane only.) WIM2 
also includes correlation modelling of large-scale parameters and scenario-dependent 
polarisation modelling [WINNER II Channel Models, 2007]. 

4.1.3.1 Model features 

1. It is a geometry-based stochastic channel model, which allows the creation of an 
arbitrary double-directional model. The channel impulse response is the sum of 
specular components. This is the same principle as used in the SCM and SCME 
channel model. The channel parameters are determined randomly, based on 
probability distributions extracted from channel measurement. 

2. It covers 12 scenarios which is a larger number than SCM or SCME. These 
include indoor environments. (SCM and SCME include only outdoor 
environment.) Each scenario allows LOS or NLOS conditions. The model 
supports mobile, nomadic and fixed systems. 

3. It allows transitions between different propagation conditions, the most 
important of which are transitions between LOS and NLOS within the same 
scenario. In the Al (indoor) or Bl (urban microcell) scenarios, for example, 
transitions from LOS to NLOS can occur as a result of the MS turning a corner to 
leave a corridor or street in which the BS is located. 

4. It can be used to characterise systems operating in the 2-6 GHz frequency range 
with a bandwidth up to 100 MHz. Like SCME, intra-cluster delay spread is used 
to support this bandwidth. 

5. It specifies DOA and DOD as two-dimensional variables (whereas only a single 
(azimuthal) angle is considered in SCM and SCME). 

6. It employs a sophisticated correlation model to the multiple links in system level 
simulations [M. Narandzic et. al., 2007]. (This contrasts with SCM which adopts a 
fixed correlation (0.5) for slow fading to model multiple links between a MS and 
different BSs.) 

7. A drop, or channel segment, represents a quasi-stationary period during which 
the probability distributions of most channel parameters do not change. The 
advantage of this approach is simplicity. The disadvantage is that it is not 
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possible to adequately simulate cases where variable channel conditions are 
needed. Drop-based simulation is the principal approach used by both WINNER 
I (SCM and SCME) and WIM2 models. WIM2, however, also allows simulation 
with time evolution in which the drops are correlated and a smooth transition 
between consecutive drops is engineered. (The drops in WIM2 are usually 
referred to as channel segments since they are no longer 'dropped in' at random.) 
This smooth transition between channel segments is realised by spacing the 
segments in time by the quasi-stationary duration and dividing the transition 
region into a number of sub-intervals. The number of sub-intervals is chosen to 
be the same as the higher of the number of clusters in either channel segment. In 
each sub-interval the strength (amplitude) of a cluster from the earlier segment is 
ramped linearly down and the strength of the cluster from the later segment is 
ramped up. The pair of clusters is chosen to be, as far as possible, 'matched' in 
strength. Where the number of clusters in earlier and later segments are not equal 
then the weakest paths in the segment with more clusters are left unpaired and 
are ramped up or down alone. This process is illustrated schematically in Fig. 13. 
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Fig. 13. Transition between channel segments by power ramping up and down of clusters 
(After [WINNER II Channel Models, 2007]). 



For time division duplex (TDD) systems WIM2 (like SCM and SCME) uses the 
same parameters for both uplink and downlink. For frequency division duplex 
(FDD) systems a different path-loss is used on uplink and downlink and the 
random phases of sub-paths on uplink and downlink are modelled 
independently [M. Narandzic et. al., 2007]. 



4.1.3.1 Channel modelling approach 

The modelling process can be divided into three phases. These are illustrated in Fig. 14. 
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Fig. 14. WIM2 channel modelling process. 

1. The environment, network layout and antenna array parameters are defined. 

2. The large-scale parameters such as slow fading (shadowing), power, DODs, 
DOAs and delay-spread are drawn randomly from tabulated distribution 
functions. At this stage the geometry (i.e. network layout) is fixed and the only 
free variables are the random initial phases of the sub-paths. By picking different 
initial phases (randomly), an unlimited number of different realisations of the 
model can be generated. When the initial phases are fixed, the model is 
deterministic. 

3. Fast fading samples are generated using the sum of sinusoids technique. (This is 
the same as in SCM and SCME.) The model implements time evolution 
(depending on user input) in order to generate correlated channel samples if the 
required channel simulation period is longer than the quasi-stationery period of 
the channel. 



4.2 WiMAX 

IEEE working group 802.16 has been central to the development of technical standards for 
fixed wireless access networks. Broadband wireless access (BWA) technology provides last 
mile access for high-speed residential and commercial Internet services. It is a promising 
alternative to digital subscriber line (DSL), cable and fibre technologies which are struggling 
to meet world-wide demand, especially outside metropolitan centres, for Internet services at 
reasonable cost. The IEEE 802.16 standard for BWA and its associated industry consortium, 
the WiMAX forum, has the potential to offer broadband access to virtually all users 
irrespective of location. WiMAX (the Worldwide Interoperability for Microwave Access) is a 
consortium of telecommunication equipment manufacturers, vendors and service providers, 
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formed to promote the compatibility and interoperability of BWA devices incorporating the 
IEEE 802.16 and ETSI HiperMAN wireless standards. 

IEEE 802.16 was designed for LOS links operating at carrier frequencies between 10 and 66 
GHz. The first release of the standard (IEEE 802.16-2001) specifies a set of medium access 
control (MAC) and physical-layer standards intended to provide fixed broadband access 
using a point-to-point (PP) or point-to-multipoint (PMP) topology. The standard was 
revised in January 2003 to included NLOS links operating at frequencies in both licensed 
and unlicensed bands between 2 and 11 GHz. A consolidated standard, IEEE 802.16-2004, 
was issued in 2004. 

IEEE 802.16e-2005, was issued in December 2005 which includes enhancements for physical 
and MAC layers that support nomadic and mobile operation in 2 to 11 GHz range. 

The WiMAX forum has adopted the IEEE 802.16-2004 and ETSI HyperMAN standards for 
fixed and nomadic access and the IEEE 802. 16e standard for portable access. 

Two channel models are used for fixed and portable systems complying with the IEEE 
802.16 standard. The Stanford University Interim (SUI) channel model is used for fixed 
broadband access and the ITU Tapped-Delay-Line channel model is used for portable 
broadband access. 



4.2.1 SUI 

The Stanford University Interim (SUI) suite of channel models was designed for fixed 
macrocell networks operating at 2.5 GHz. It contains the definition of six specific channel 
implementations which were initially developed for Multipoint Microwave Distribution 
Systems (MMDSs) in the USA operating in the 2.5 - 2.7 GHz frequency band. Their 
applicability to the 3.5 GHz frequency band that is in use in the UK has so far not been 
conclusively established. 

The model generates both SISO and MIMO channel parameters. The six specific channel 
implementations represent different terrain types. The targeted scenarios are based on the 
following assumptions: 

1. Cell radius less than 10 km. 

2. Cell coverage (80 - 90%). 

3. Fixed directional receiving antenna installed at a height of 2 - 10 m (below 
rooftop height since LOS is not required). 

5. Base-station antenna installed at a height of 15 - 40 m (above rooftop height). 

6. Flexible channel bandwidth between 2 and 20 MHz. 

Three terrain types are defined: A for hilly terrain with moderate to heavy tree density 
(representing the highest path-loss terrain type), B for either flat terrain with moderate to 
heavy tree density or hilly terrain with light tree density and C for flat terrain with 
moderate to light tree density (representing the lowest path-loss terrain). 
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The SUI models corresponding to these terrain types are: 

1. A: SUI-5 and SUI-6 

2. B: SUI-3 and SUI-4 

3. C: SUI-1 and SUI-2 
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Fig. 15. Tapped delay line channel model. (After[T. S. Rapport, 2002]) 



4.2.1.1 Model features 

1. Claimed to be valid between 2 and 4 GHz (although no tests of its performance in 
the European 3.5 GHz band are known) and for up to 7 km separation between 
transmitter and receiver. 

2. Fading modelled as tapped delay line, Fig. 15, with three taps having non- 
uniform delays. The delays are specified in the standard document [IEEE 802.16 
(BWA), 2003]. 

3. Omni-directional antennas are assumed at both transmitter and the receiver in 
the original SUI models. A modified version of the model assumes directional 
antennas with 30° beam widths. 

4. A lognormal model [L.J. Greenstein et. AL, 1999] for the distribution of K-factor is 
adopted. The K-i actor for each tap is specified by exceedance values 
corresponding to cell coverage areas of 90% and 75% (SUI-1 to SUI-4) and 90%, 
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10. 



75% and 50% (SUI-5 to SUI-6). Taps 2 and 3 are always, effectively, Rayleigh 
faded. Tap 1 may be Ricean or Rayleigh faded. 

Correlation between multipath components (tap weights) for the same taps of 
different receiving antennas at the MS is fixed irrespective of MS array 
configuration. The correlation between multipath components for the same taps 
of different receiving antennas at the BS is zero (corresponding to an assumption 
of large antenna spacing. 

Channel coefficients with the probability distribution and power spectral density 
specified in the standard document [IEEE 802.16 (BWA), 2003] are generated by 
filtering noise. In a fixed wireless system the Doppler power spectral density is 
concentrated around / = Hz. The shape of the spectrum, which different from 
the classical Jakes spectrum (that is more relevant to urban mobile scenarios), is 
given by: 

\l.72(f D /f m ) 2 + 0JS5(f D /fS f D <f m ] 

S(f D ) = \ n , , (8) 

where f m is the maximum Doppler frequency. This is usually referred to as the 

'rounded' Doppler spectrum [IEEE 802.16 (BWA), 2003]. 

Validity restricted to distances less than 7 km (which is less than the targeted 

maximum range of 10 km). A modified version of SUI channel model has been 

proposed in order to scale transmit-receive distances to ranges greater than 7 km. 

Propagation environments characterised in terms of power delay profiles. 

Actual antenna array configurations not considered. 

Simple and suited for rapid proto-typing or equipment/ algorithm development. 

Unsuited to comprehensive, location-specific, network/ system planning. 



5.2.1.2 Generic structure 

The generic structure of the SUI channel model is shown in Fig. 16 [IEEE 802.16 (BWA), 
2003]. 
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Fig. 16. Generic structure of SUI channel model. (After [IEEE 802.16 (BWA), 2003].) 



The input mixing matrix models the spatial correlation between inputs if multiple antennas 
are used at the transmitter. The tapped delay line matrix models multipath fading. Each 
filter tap is characterised by a Ricean or Raleigh fading process. The output mixing matrix 
models the spatial correlation between output signals if multiple antennas are used at the 
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5.2.1.3 Channel modelling approach 

1. RMS delay-spread is calculated based on user specified parameters. 

2. A set of complex zero-mean Gaussian random numbers are generated for each 
tap leading to a Rayleigh distributed tap magnitude. If a Ricean distribution (K > 
0) is required a constant path component is added to the Rayleigh set of 
coefficients. 

3. The random numbers generated are independent (and therefore uncorrelated) 
and thus have a white spectrum. The SUI channel model specifies a 'rounded' 
power spectrum. The uncorrelated samples are filtered to generate channel 
coefficients with the required correlation properties. 

4. An antenna envelope correlation value (i.e. the correlation between the amplitude 
of signals received at corresponding taps of two antenna elements) is defined for 
multiple transmit or receive elements. 

The simulation steps are summarised in Fig. 17. 
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Fig. 17. SUI channel modelling process 



4.2.2 WiMAX- ITU-TDL 

The WiMAX forum approved the mobile WiMAX system profile in 2006. Mobile WiMAX, 
based on 802.16e-2005, enables WiMAX systems to address portable and mobile devices in 
addition to fixed and nomadic applications. The WiMAX forum Mobile release 1.0 channel 
model [WiMAX forum, 2008] defines the SISO and MIMO channel model requirements for 
mobile applications governed by the IEEE 802.11e standard. The purpose of the model is to 
provide a realistic and repeatable channel context for the testing and comparison of portable 
and mobile WiMAX-enabled devices. 
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4.2.2.1 Model features 

Mobile WiMAX specifies SISO/MIMO channel models for radio conformance testing (RCT) 
of WiMAX products. The WiMAX forum has selected the ITU Pedestrian-B and Vehicular- A 
6-tap TDL models for test and verification of SISO devices. The WiMAX-ITU-TDL channel 
model extends the ITU-TDL model to the MIMO systems encompassing spatial correlation 
for each multipath component. One of the novel features of the model is the definition of 
three levels of channel correlation (low, medium and high) between antenna elements. The 
following are the main features of the model: 

1. It is a physical model, using a stochastic modelling approach similar to SCM. 

2. ITU propagation scenarios are extended by defining the azimuth angle spread 
and shape (Laplacian) of the azimuth spectrum for each tap. Tap-wise MIMO 
correlation matrices are determined based on the spatial information (i.e. high, 
medium and low antenna correlation) of multipath components combined with 
specific antenna configurations. 
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Fig. 18. Three levels of MIMO correlation for WiMAX-ITU-TDL. (After [WiMAX forum, 
2008].) 

3. Three pairs of reference antenna configurations are defined that typify low, 
medium and high correlation. The reference configurations specify the spacing of 
elements and their relative polarisations at MS and BS. This is illustrated 
schematically in Fig. 18. 

4. The azimuth spread in all WiMAX-ITU-TDL scenarios for all taps is 2° at the BS 
and 35° at the MS. 

5. The Doppler spectrum has the classical Jakes U-shape [Jakes model]. 

6. The strength of each tap is Raleigh distributed. 

7. To model long channel impulse response (>10 us) and for high-speed MSs (>120 
km/h), the ITU Vehicular A channel is used but modified such that the last tap is 
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moved from 2510 ns to 10,000 ns. (The strength of the last tap remains unchanged 
at -20dB.) 
8. Each cluster corresponds to a resolvable multipath. Within each of the six 

multipaths there are 20 irresolvable sub-paths. Each of the six multipaths is 
modelled as Dirac (delta) function of delay. The 20, spatially separated, sub-paths 
have equal delay. 

4.2.2.2 WiMAX-ITU-TDL model design 

1. The model can be separated into three components addressing (i) spatial 
correlation, (ii) polarisation correlation and (iii) temporal correlation. The spatial 
and polarisation correlation are modelled using square matrices of dimensions 
(KxM*N)x(KxMxN) where K is the number of multipaths, and M and N are the 
number of elements in the transmit and receive antenna arrays respectively. The 
temporal correlation matrix has same number of rows as the spatial or 
polarisation matrices and the same number of columns as the number of required 
channel samples. 

2. Spatial correlation is calculated independently at transmit and receive antenna 
and is based on antenna geometry. A 2x2 MIMO system results in the following 
spatial correlation matrices at transmit and receive antenna elements: 
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where a and ft are the spatial correlations between antenna elements at BS and MS 
respectively. The MIMO correlation matrix is given by the Kronecker product of 
independent spatial correlation matrices, i.e.: 
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The use of the Kronecker product corresponds to an assumption that the 
correlation between elements in the received antenna array is not affected by 
changes in the spatial configuration of elements in the transmit antenna array. 
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4. The polarisation model (which originated in the ITU) assumes a cross-polar ratio 

(XPR = cross-polar power/ co-polar power) of -8 dB for each tap [WiMAX forum, 
2008]. The polarisation matrix is denoted by: 



s = 



^3 vv *3 vh 
jShv i$hh m 
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where v and h denote vertical and horizontal polarisation respectively. (The first 
subscript refers to the BS and the second subscript refers to the MS.) The 
downlink polarisation correlation matrix for a 2x2 MIMO system [WiMAX 
forum, 2008] is given by: 
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where y depends on XPR (expressed as a power ratio). The structure of Rp,mimo 
depends on the MS and BS antenna polarisations and orientations. Further detail 
can be found in [WiMAX forum, 2008]. 
5. Temporal correlation refers to the auto-correlation of the signal received at a 

single antenna element. There are two ways of generating temporal (fast fading) 
tap coefficients. Since the Doppler spectrum is classical and the amplitude is 
Rayleigh distributed the temporal fading samples can be generated using either 
the sum of sinusoids technique (Jakes' method) or by filtering white Gaussian 
noise. Fig. 19 summarises the process of generation of channel weights. 
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Fig. 19. Block diagram of correlated channel tap generation. (After [WiMAX forum, 2008].) 



4.2.2.3 Software implementation of WiMAX-ITU-TDL 

Fig. 20 shows pseudo-code for a WiMAX-ITU-TDL channel model. The modelling process is 
divided into the following principal components: 
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r l_s.p_mimci Is the Square Matrix that Stores Rj p.wmo Matrix for each MIMD Link 



Rmhh Is the matrix product of R l _*j>_mim* and Riim* 

Finally the fading samples are scaled by the path 

loss and shadow fading) 



Generate Temporal Correlation Samples (Ri*™,) 

(fast fading samples are stored in a matrix of size 

[No of Tx * Rx >MuiliprtLlii| * [TiimSamplcsl 



Fig. 20. Pseudo-code flow diagram for WiMAX-ITU-TDL channel model. 
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The generation of secondary channel statistics (e.g azimuth spread, offset DO A and offset 
DOD) is based on the user specified scenario. The channel segment (drop) represents a 
quasi-stationary period in which the probability distributions of parameters are unchanged. 
Unlike SCME the underlying large-scale parameters such as DOD and DOA remain 
constant within a single a channel segment. 

The spatial correlation matrices for the BS and MS antenna elements are generated for all 
multipaths of each link. The BS and MS antenna spatial correlation matrices are Kronecker 
multiplied. The spatial correlation matrix remains unchanged between multiple drops of the 
same simulation. 

Like spatial correlation the polarisation matrix remains unchanged between multiple drops 
of the same simulation. 

Temporal correlation refers to the autocorrelation of the signal received at a single antenna 
element. During each drop of the simulation fast fading samples of the tap weights are 
generated either by the summing sinusoids or filtering a white Gaussian random process. 

4.3 IEEE 802.1 1n 

IEEE 802.11n is an amendment to the IEEE 802.11-2007 wireless local area network (WLAN) 
standard [IEEE 802.11, 2009]. It gives higher network throughput compared with earlier 
variants of the standard, e.g. 802.11b and 802.11g. The maximum raw (physical-layer) data 
rate is 600 Mbit/s. The IEEE 802.11n channel model [V. Erceg et. al., 2004] was developed for 
systems using the 2 GHz and 5 GHz bands operating in indoor environments and 
employing MIMO technology. Measurement results from these two frequencies bands were 
combined to develop the model. Only the path-loss element of the model depends on 
frequency. The channel model is based on the clustering approach developed by Saleh and 
Valenzula [A. A. M. Saleh et. al, 1987] and illustrated in Fig. 21. 

4.3.1 Model features 

The principal features of the model [P. Aimer et. al., 2007] are: 

1. It is a physical model based on stochastic modelling approach. 

2. It can be applied to both 2 GHz and 5 GHz frequency bands. 

3. A dual-slope, frequency dependent, model for path-loss is adopted. 

4. The channel impulse response is a sum of clusters where each cluster consists of up 
to 18 multipaths. (The precise number depends on the scenario.) These multipaths 
are modelled as filter taps which are separated in delay by at least 10 ns. 

5. Five scenarios are defined [P. Aimer et. al., 2007]: A, B, C, D, E and F. Scenarios 
A-C represent small environments (RMS delay-spread <30 ns), such as 
residential homes and small offices. (Scenario A is optional and not 
recommended for system performance comparison.) Scenarios D-F represent 
large open spaces with maximum RMS delay-spread <150 ns. 
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1/A 



-T/r 



-tir 



A = Cluster Arrival Rate 
y u = Ray Arrival Rate 

= Power Decay Constant for Cluster 
V = Power Decay Constant for Ray 




Fig. 21. Example channel impulse response derived from a Saleh and Valenzuela channel 
model. (After [A. A. M. Saleh et. al.].) 

6. The MIMO channel matrix H for each filter tap at each instant of time is 

separated into a fixed (constant) LOS matrix, Hf, and a time-varying (Rayleigh 
distributed) NLOS matrix, H v [V. Erceg et. al, 2004], i.e.: 



H 



K 



K + \ 



H, 



1 



K + \ 



-H, 



13 



where K is the Ricean K-factor. 



7. Each tap consists of a number of sub-paths having truncated Laplacian azimuth 
spectrum with an angular spread that varies between from 20° to 40° [V. Erceg et. 
al.,2004]. 

8. Elevation spread is not incorporated (since most building dimensions in azimuth 
are much larger than their dimensions in elevation) [V Erceg et. al., 2004]). 

9. The mean DO A and DOD for each cluster is random with a uniform distribution 
over all azimuth angles. (For indoor WLANs the scattering environment is 
similar for both the access point and the user equipment. This is in contrast to 
outdoor scenarios where the BS is typically mounted sufficiently high to be 
relatively free of local scatter while the MS is often immersed in a rich scattering 
environment.) 

10. Several channel taps in scenarios D and E are amplitude modulated to account 
for the effects of fluorescent lights [V. Erceg et. al., 2004] which represent time- 
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11. 

12. 



periodic scatterers; alternately present and absent at twice the frequency of the 
mains supply (2x50 Hz in Europe). 

The model differentiates between uplink and downlink (unlike SCM and SCME). 
Isotropic antenna elements are assumed. 



User Specified Parameters 



T 



Propagation Parameters 

PAS Type, AS T *, AS R *. DoD, DoA 
Path Loss and Shadowing 



Construct Transmit Correlation 
Matrix (R T *) 



Construct Receive Correlation 
Matrix (Rr*) 



Spatial Correlation 



Kronecker Product 

Rt* & Rr* (Downlink) 
Rr*0 Rt* (Uplink) 



H iid 



IIR Filtering 



X = {[RrJ <g> [Ra 1 ' 2 [HJ 

BE 



i NLOS Component 



Temporal Correlation 



LOS Component 



e- 



Ricean Phase 
Generation (H F ) 



Add Fluorescent Effect 



Add Path Loss and 
Shadowing 



Fig. 22. IEEE 802.11n channel model 

4.3.2 Model features 

The various steps in the IEEE 802.11n channel model are illustrated in Fig. 22 and described 
below: 



1. At each drop, during the initialisation phase, the model determines the delay 
profile and identifies clusters in the user-specified scenario. 

2. The model assigns azimuth spread, mean DOD and DOA to each cluster and the 
corresponding taps. Spatial correlation is calculated independently at transmit 
and receive antennas and is based on antenna geometry. 

3. The spatial correlation matrix is the Kronecker product of the individual spatial 
correlation matrices of transmit and receives antenna arrays, Fig. 22. The spatial 
correlation is a square matrix and is different for uplink and downlink. 
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4. Fast fading samples are generated by filtering uncorrected (white) Gaussian 
noise. The filter final states are retained after a dummy run in order to avoid 
transient states. (Fading samples generated during the transient phase are not 
used because their variance is artificially low.) 

5. In order to maintain continuity between successive channel calls (for the same 
drop) the filter states are stored. 

6. The product of spatial and temporal correlation matrices results in the H matrix 
which is then scaled to account for path-loss and shadowing. 

5. Summary and comparison of channel models 

Table 1 summarises the principal features of those channel models that have been described 
and Table 2 compares some of their most important parameters. 





SCM 


SOME 


WINNER II 


SUI 


WiMAX 


IEEE 802.1 1n 


Bandwidth > 100 MHz 


No 


Yes* 


Yes 


No 


No 


Yes 


Indoor scenarios 


No 


No 


Yes 


No 


No 


Yes ** 


Outdoor-to-indoor and 
indoor-to-outdoor scenarios 


No 


No 


Yes 


No 


No 


No 


DoD/DoA elevation 


No 


No 


Yes 


No 


No 


No 


Random selection of MPC from the 
appropriate probability distribution 


Yes 


Yes 


Yes 


No 


No 


No 


Time evolution of large scale model 
parameters with in a drops 


No 


Yes 


Yes 


No 


No 


No 


Time evolution of large scale model 
parameters between successive drops 


No 


No 


Yes 


No 


No 


No 


Use of Intra-Cluster delay spread concept 
(based on Saleh and Valenzuela model 


No 


Yes 


Yes 


No 


No 


Yes 


Cross correlation between large scale 
parameters 


No 


No 


Yes 


No 


No 


No 



SCME support both 20 MHz and 100 MHz bandwidth 

IEEE802.1 1n is an indoor channel model with only one 
case that typical open space (both indoor and outdoor) 



Table 1. Principal features of standard channel models. (Extended from [M. Narandzic et. 
al, 2007].) 
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Unit 


SCM 


SCME 


WINNER II 


SUI 


WiMAX 


IEEE 802.1 1n 


Max bandwidth 


MHz 


5 


100* 


100** 


2-20 


2 


100 


Frequency range 


GHz 


2 


2-6 


2-6 


1-4 


2-11 


2-5 


No. of scenarios 




3 


3 


12 


6 


3 


6 


No. of clusters 




6 


6 


4-20 





6 


2-6 


No. of taps 




6 


18-24 


4-24 


3-4 


6 


1-18 


Doppler Spectrum 




Classical 


Classical 


Classical 


Standard 
Specific 


Classical 


Standard 
Specific 


Normalised Delay spread 


MS 


1.2 


1.2 


1.2 


0.9-20 


2.51-10 


0-1.05 


Path azimuth spread 
at BS and MS 


Degrees 


2,5,35 


2,5,35 


5-50*** 


NA 


2,35 


20-40 


Shadow fading 
(standard deviation) 


dB 


4-10 


4-10 


3-8 


8.5-10.5 


8-12 


3-6 



* Artificial extension from 5 MHz bandwidth 

** Based on 100 MHz measurements 

*** Per tap azimuth spread in WINNER varies between 5°-50° depending on the scenario 
**** The combined shadow fading/GRF standard deviation a c can be calculated using standard defined formula. 
The value represents a special case of 20° antenna beam width 



Table 2. Comparison of standard channel model parameters. (Extended from [M. Narandzic 
et. al, 2007].) 

6. Summary 

Next generation wireless systems will offer wide bandwidth, high data-rates and greater 
mobility. MIMO technology will certainly play an important role in future wireless 
application. This chapter has presented a brief review of the theoretical framework used to 
describe MIMO channels and has described a selection of standard MIMO channel models. 
The characteristics of standard channel models have been summarised and compared. 
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1. Introduction 

Usually, the purpose of studying data compression algorithms is twofold. The need for effi- 
cient storage and transmission is often the main motivation, but underlying every compres- 
sion technique there is a model that tries to reproduce as closely as possible the information 
source to be compressed. This model may be interesting on its own, as it can shed light on the 
statistical properties of the source. DNA data are no exception. We urge to find out efficient 
methods able to reduce the storage space taken by the impressive amount of genomic data 
that are continuously being generated. Nevertheless, we also desire to know how the code of 
life works and what is its structure. Creating good (compression) models for DNA is one of 
the ways to achieve these goals. 

Recently, and with the completion of the human genome sequencing, the development of effi- 
cient lossless compression methods for DNA sequences gained considerable interest (Behzadi 
and Le Fessant, 2005; Cao et al, 2007; Chen et al, 2001; Grumbach and Tahi, 1993; Korodi and 
Tabus, 2005; 2007; Manzini and Rastero, 2004; Matsumoto et al., 2000; Pinho et al., 2006; 2009; 
2008; Rivals et al., 1996). For example, the human genome is determined by approximately 
3 000 million base pairs (Rowen et al., 1997), whereas the genome of wheat has about 16 000 
million (Dennis and Surridge, 2000). Since DNA is based on an alphabet of four different sym- 
bols (usually known as nucleotides or bases), namely, Adenine (A), Cytosine (C), Guanine (G), 
and Thymine (T), without compression it takes approximately 750 MBytes to store the human 
genome (using log 2 4 = 2 bits per symbol) and 4 GBytes to store the genome of wheat. 
In this chapter, we address the problem of DNA data modeling and coding. We review the 
main approaches proposed in the literature over the last fifteen years and we present some 
recent advances attained with finite-context models (Pinho et al., 2006; 2009; 2008). Low-order 
finite-context models have been used for DNA compression as a secondary, fall back method. 
However, we have shown that models of orders higher than four are indeed able to attain 
significant compression performance. 

Initially, we proposed a three-state finite-context model for DNA protein-coding regions, i.e., 
for the parts of the DNA that carry information regarding how proteins are synthesized (Fer- 
reira et al., 2006; Pinho et al., 2006). This three-state model proved to be better than a single- 
state model, giving additional evidence of a phenomenon that is common in these protein- 
coding regions, the periodicity of period three. 
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More recently (Pinho et al., 2008), we investigated the performance of finite-context models 
for unrestricted DNA, i.e., DNA including coding and non-coding parts. In that work, we 
have shown that a characteristic usually found in DNA sequences, the occurrence of inverted 
repeats, which is used by most of the DNA coding methods (see, for example, Korodi and 
Tabus (2005); Manzini and Rastero (2004); Matsumoto et al. (2000)), could also be successfully 
integrated in finite-context models. Inverted repeats are copies of DNA sub-sequences that 
appear reversed and complemented (A o T,C <-)• G) in some parts of the DNA. 
Further studies have shown that multiple competing finite-context models, working on a 
block basis, could be more effective in capturing the statistical information along the sequence 
(Pinho et al., 2009). For each block, the best of the models is chosen, i.e., the one that requires 
less bits for representing the block. In fact, DNA is non-stationary, with regions of low infor- 
mation content (low entropy) alternating with regions with average entropy close to two bits 
per base. This alternation is modeled by most DNA compression algorithms by using a low- 
order finite-context model for the high entropy regions and a Lempel-Ziv dictionary based 
approach for the repetitive, low entropy regions. In this work, we rely only on finite-context 
models for representing both regions. 

Modeling DNA data using only finite-context models has advantages over the typical DNA 
compression approaches that mix purely statistical (for example, finite-context models) with 
substitutional models (such as Lempel-Ziv based algorithms): (1) finite-context models lead 
to much faster performance, a characteristic of paramount importance for long sequences (for 
example, some human chromosomes have more than 200 million bases); (2) the overall model 
might be easier to interpret, because it is made of sub-models of the same type. 
This chapter is organized as follows. In Section 2 we provide an overview of the DNA com- 
pression methods that have been proposed. Section 3 describes the finite-context models used 
in this work. These models collect the statistical information needed by the arithmetic cod- 
ing. In Section 4 we provide some experimental results. Finally, in Section 5 we draw some 
conclusions. 

2. DNA compression methods 

The interest in DNA coding has been growing with the increasing availability of extensive 
genomic databases. Although only two bits are sufficient to encode the four DNA bases, 
efficient lossless compression methods are still needed due to the large size of DNA sequences 
and because standard compression algorithms do not perform well on DNA sequences. As a 
result, several specific coding methods have been proposed. Most of these methods are based 
on searching procedures for finding exact or approximate repeats. 

The first method designed specifically for compressing DNA sequences was proposed by 
Grumbach and Tahi (1993) and was named Biocompress. This technique is based on the sliding 
window algorithm proposed by Ziv and Lempel, also known as LZ77 (Ziv and Lempel, 1977). 
According to this universal data compression technique, a sub-sequence is encoded using a 
reference to an identical sub-sequence that occurred in the past. Biocompress uses a charac- 
teristic usually found in DNA sequences which is the occurrence of inverted repeats. These 
are sub-sequences that are both reversed and complemented (A o T, C o G). The second 
version of Biocompress, Biocompress-2, introduced an additional mode of operation, based on 
an order-2 finite-context arithmetic encoder (Grumbach and Tahi, 1994). 
Rivals et al. (1995; 1996) proposed another compression technique based on exact repetitions, 
Cfact, which relies on a two-pass strategy. In the first pass, the complete sequence is parsed 
using a suffix tree, producing a list of the longest repeating sub-sequences that have a potential 
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coding gain. In the second pass, those sub-sequences are encoded using references to the past, 
whereas the rest of the symbols are left uncompressed. 

The idea of using repeating sub-sequences was also exploited by Chen et al. (1999; 2001). 
The authors proposed a generalization of this strategy such that approximate repeats of sub- 
sequences and of inverted repeats could also be handled. In order to reproduce the original 
sequence, the algorithm, named GenCompress, uses operations such as replacements, inser- 
tions and deletions. As in Biocompress, GenCompress includes a mechanism for deciding if it is 
worthwhile to encode the sub-sequence under evaluation using the substitution-based model. 
If not, it falls back to a mode of operation based on an order-2 finite-context arithmetic encoder. 
A further modification of GenCompress led to a two-pass algorithm, DNACompress, relying on 
a separated tool for approximate repeat searching, PatternHunter, (Chen et al., 2002). Besides 
providing additional compression gains, DNACompress is considerably faster than GenCom- 
press. 

Before the publication of DNACompress, a technique based on context tree weighting (CTW) 
and LZ-based compression, CTW+LZ, was proposed by Matsumoto et al. (2000). Basically, 
long repeating sub-sequences or inverted repeats, exact or approximate, are encoded by a 
LZ-type algorithm, whereas short sub-sequences are compressed using CTW. 
One of the main problems of techniques based on sub-sequence matching is the time taken by 
the search operation. Manzini and Rastero (2004) addressed this problem and proposed a fast, 
although competitive, DNA encoder, based on fingerprints. Basically, in this approach small 
sub-sequences are not considered for matching. Instead, the algorithm focus on finding long 
matching sub-sequences (or inverted repeats). Like most of the other methods, this technique 
also uses fall back mechanisms for the regions where matching fails, in this case, finite-context 
arithmetic coding of order-2 (DNA2) or order-3 (DNA3). 

Tabus et al. (2003) proposed a sophisticated DNA sequence compression method based on 
normalized maximum likelihood discrete regression for approximate block matching. This 
work, later improved for compression performance and speed (Korodi and Tabus (2005), 
GeNML), encodes fixed-size blocks by referencing a previously encoded sub-sequence with 
minimum Hamming distance. Only replacement operations are allowed for editing the ref- 
erence sub-sequence which, therefore, always have the same size as the block, although may 
be located in an arbitrary position inside the already encoded sequence. Fall back modes of 
operation are also considered, namely, a finite-context arithmetic encoder of order-1 and a 
transparent mode in which the block passes uncompressed. 

Behzadi and Le Fessant (2005) proposed the DNAPack algorithm, which uses the Hamming 
distance (i.e., it relies only on substitutions) for the repeats and inverted repeats, and either 
CTW or order-2 arithmetic coding for non-repeating regions. Moreover, DNAPack uses dy- 
namic programming techniques for choosing the repeats, instead of greedy approaches as 
others do. 

More recently, two other methods have been proposed (Cao et al., 2007; Korodi and Tabus, 
2007). One of them (Korodi and Tabus, 2007), is an evolution of the normalized maximum 
likelihood model introduced by Tabus et al. (2003) and improved by Korodi and Tabus (2005). 
This new version, NML-1, is built on the GeNML framework and aims at finding the best 
regressor block using first-order dependencies (these dependencies were not considered in 
the previous approach). 

The other method, proposed by Cao et al. (2007) and called XM, relies on a mixture of ex- 
perts for providing symbol by symbol probability estimates which are then used for driv- 
ing an arithmetic encoder. The algorithm comprises three types of experts: (1) order-2 
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Markov models; (2) order-1 context Markov models, i.e., Markov models that use statis- 
tical information only of a recent past (typically, the 512 previous symbols); (3) the copy 
expert, that considers the next symbol as part of a copied region from a particular off- 
set. The probability estimates provided by the set of experts are them combined using 
Bayesian averaging and sent to the arithmetic encoder. Currently, this seems to be the method 
that provides the highest compression on the April 14, 2003 release of the human genome 
(see results in ftp://ftp.infotech.monash.edu.au/software/DNAcompress-XM/ 
XMCompress/humanGenome .html). However, both NML-1 and XM are computationally 
intensive techniques. 

3. Finite-context models 

Consider an information source that generates symbols, s, from an alphabet A. At time t, the 
sequence of outcomes generated by the source is x l = x\%i . . . xt. A finite-context model of 
an information source (see Fig. 1) assigns probability estimates to the symbols of the alphabet, 
according to a conditioning context computed over a finite and fixed number, M, of past 
outcomes (order-M finite-context model) (Bell et al., 1990; Salomon, 2007; Sayood, 2006). At 
time t, we represent these conditioning outcomes by c 1 = x t _M+i' • • • / **-l/ *t- The number of 
conditioning states of the model is \A\ M , dictating the model complexity or cost. In the case 
of DNA, since \A\ = 4, an order-M model implies 4 M conditioning states. 



Xt-A 

I G I A I A I T 



x t +i 

aig| a|c I c I t |g| 




P(x t+1 = s\<?) 



Encoder 

Output 

bit-stream 
Fig. 1. Finite-context model: the probability of the next outcome, x t +i, is conditioned by the 
M last outcomes. In this example, M = 5. 



In practice, the probability that the next outcome, Xf+i, is s, where s £ A = {A, C, G, T}, is 
obtained using the Lidstone estimator (Lidstone, 1920) 



P(*t+l 



nj + S 

aeA 



(i) 



where n\ represents the number of times that, in the past, the information source generated 
symbol s having c l as the conditioning context. The parameter 5 controls how much probabil- 
ity is assigned to unseen (but possible) events, and plays a key role in the case of high-order 
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Context, c 1 


n\ 


4 


n' G 


tij 




AAAAA 


23 


41 


3 


12 


79 


ATAGA 


16 


6 


21 


15 


58 


GTCTA 


19 


30 


10 


4 


63 


TTTTT 


8 


2 


18 


11 


39 



Table 1. Simple example illustrating how finite-context models are implemented. The rows 
of the table represent probability models at a given instant t. In this example, the particular 
model that is chosen for encoding a symbol depends on the last five encoded symbols (order-5 
context). 



models. 1 Note that Lidstone's estimator reduces to Laplace's estimator for 8 = 1 (Laplace, 
1814) and to the frequently used Jeffreys (1946) / Krichevsky and Trofimov (1981) estimator 
when 8 = 1/2. In our work, we found out experimentally that the probability estimates cal- 
culated for the higher-order models lead to better compression results when smaller values of 
8 are used. 

Note that, initially, when all counters are zero, the symbols have probability 1/4, i.e., they are 
assumed equally probable. The counters are updated each time a symbol is encoded. Since 
the context template is causal, the decoder is able to reproduce the same probability estimates 
without needing additional information. 

Table 1 shows an example of how a finite-context model is typically implemented. In this 
example, an order-5 finite-context model is presented (as that of Fig. 1). Each row represents a 
probability model that is used to encode a given symbol according to the last encoded symbols 
(five in this example). Therefore, if the last symbols were "ATAGA" , i.e., c l = ATAGA, then 
the model communicates the following probability estimates to the arithmetic encoder: 

P(A\ ATAGA) = (16 + ^/(58 + 4^), 



P(C\ATAGA) 
P(G\ATAGA) : 



(6 + <5)/(58 + 4<S), 
(21 + J)/ (58 + 48) 



and 



P(T\ ATAGA) = (15 + £)/(58 + 4J). 
The block denoted "Encoder" in Fig. 1 is an arithmetic encoder. It is well known that practical 
arithmetic coding generates output bit-streams with average bitrates almost identical to the 
entropy of the model (Bell et al, 1990; Salomon, 2007; Sayood, 2006). The theoretical bitrate 
average (entropy) of the finite-context model after encoding N symbols is given by 



N-l 



Hk 



N 



E lo S2 p 0m = s k*) b P s ' 



(2) 



t=o 



1 When M is large, the number of conditioning states, 4 M , is high, which implies that statistics have to be 
estimated using only a few observations. 
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Context, ^ 


"*A 


4 


< 


Tij 


aeA 


AAAAA 


23 


41 


3 


12 


79 


ATAGA 


16 





21 


15 


59 


GTCTA 


19 


30 


10 


4 


63 


TTTTT 


8 


2 


18 


11 


39 



Table 2. Table 1 updated after encoding symbol "C" , according to context "ATAGA" . 

where "bps" stands for "bits per symbol". When dealing with DNA bases, the generic 
acronym "bps" is sometimes replaced with "bpb", which stands for "bits per base". Recall 
that the entropy of any sequence of four symbols is, at most, two bps, a value that is achieved 
when the symbols are independent and equally likely. 

Referring to the example of Table 1, and supposing that the next symbol to encode is "C", 
it would require, theoretically, — log 2 ((6 + J)/ (58 + 4£)) bits to encode it. For 8 = 1, this is 
approximately 3.15 bits. Note that this is more than two bits because, in this example, "C" 
is the least probable symbol and, therefore, needs more bits to be encoded than the more 
probable ones. After encoding this symbol, the counters will be updated according to Table 2. 

3.1 Inverted repeats 

As previously mentioned, DNA sequences frequently contain sub-sequences that are reversed 
and complemented copies of some other sub-sequences. These sub-sequences are named "in- 
verted repeats". As described in Section 2, this characteristic of DNA is used by most of the 
DNA compression methods that rely on the sliding window searching paradigm. 
For exploring the inverted repeats of a DNA sequence, besides updating the corresponding 
counter after encoding a symbol, we also update another counter that we determine in the 
following way. Consider the example given in Fig. 1, where the context is the string "ATAGA" 
and the symbol to encode is "C". Reversing the string obtained by concatenating the context 
string and the symbol, i.e., " AT AG AC" , we obtain the string "C AG AT A". Complementing 
this string (A +-> T,C «-)■ G), we get "GTCTAT". Now we consider the prefix "GTCTA" as the 
context and the suffix "T" as the symbol that determines which counter should be updated. 
Therefore, according to this procedure, for taking into consideration the inverted repeats, after 
encoding symbol "C" of the example in Fig. 1, the counters should be updated according to 
Table 3. 



3.2 Competing finite-context models 

Because DNA data are non-stationary, alternating between regions of low and high entropy, 
using two models with different orders allows a better handling both of DNA regions that are 
best represented by low-order models and regions where higher-order models are advanta- 
geous. Although both models are continuously been updated, only the best one is used for 
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Context, c l 


n\ 


4 


4 


Hj 




AAAAA 


23 


41 


3 


12 


79 


ATAGA 


16 





21 


15 


59 


GTCTA 


19 


30 


10 


a 


64 


TTTTT 


8 


2 


18 


ii 


39 



Table 3. Table 1 updated after encoding symbol "C" according to context "ATAGA" (see 
example of Fig. 1) and taking the inverted repeats property into account. 



encoding a given region. To cope with this characteristic, we proposed a DNA lossless com- 
pression method that is based on two finite-context models of different orders that compete 
for encoding the data (see Fig. 2). 

For convenience, the DNA sequence is partitioned into non-overlapping blocks of fixed size 
(we have used one hundred DNA bases), which are then encoded by one (the best one) 
of the two competing finite-context models. This requires only the addition of a single bit 
per data block to the bit-stream in order to inform the decoder of which of the two finite- 
context models was used. Each model collects statistical information from a context of 
depth Mf,i = 1,2, Mi ^ M2. At time t, we represent the two conditioning outcomes by 
4 = z*-Mi+l/ • • • , **-i, x t and by c£ = *t-M 2 +l/ • • • / x t-l> x t- 




Fig. 2. Proposed model for estimating the probabilities: the probability of the next outcome, 
x t+ i, is conditioned by the Mi or M2 last outcomes, depending on the finite-context model 
chosen for encoding that particular DNA block. In this example, Mi = 5 and M2 = 11. 
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Using higher-order context models leads to a practical problem: the memory needed to repre- 
sent all of the possible combinations of the symbols related to the context might be too large. In 
fact, as we mentioned, each DNA model of order-M implies 4 M different states of the Markov 
chain. Because each of these states needs to collect statistical data that is necessary to the en- 
coding process, a large amount of memory might be required as the model order grows. For 
example, an order-16 model might imply a total of 4294 967296 different states. 





Hash table 






Key 


Model 




1 




P(xt+i = s|4) 

Input 
symbol 




Hash 
function 








A r t 






1 


GTTGAGCAATAGACCTG ••• 



x t -io x t+i 

Fig. 3. The context model using hash tables. The hash table representation is shown in Fig. 4. 

In order to overcome this problem, we implemented the higher-order context models using 
hash tables. With this solution, we only need to create counters if the context formed by the 
M last symbols appears at least once. In practice, for very high-order contexts, we are limited 
by the length of the sequence. In the current implementation we are able to use models of 
orders up to 32. However, as we will present later, the best value of M for the higher-order 
models is 16. This can be explained by the well known problem of context dilution. Moreover, 
for higher-order models, a large number of contexts occur only once and, therefore, the model 
cannot take advantage of them. 

For each symbol, a key is generated according to the context formed by the previous symbols 
(see Fig. 3). For that key, the related linked-list if traversed and, if the node containing the 
context exists, its statistical information is used to encode the current symbol. If the context 
never appeared before, a new node is created and the symbol is encoded using an uniform 
probability distribution. A graphical representation of the hash table is presented in Fig. 4. 
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Fig. 4. Graphical representation of the hash table used to represent higher-order models. Each 
node stores the information of the context found (Context) and the counters associated to 
that context (Counters), four in the case of DNA sequences. 
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4. Experimental results 

For the evaluation of the methods described in the previous section, we used the same DNA 
sequences used by Manzini and Rastero (2004), which are available from www . mf n . unipmn . 
it/~manzini/dnacorpus. This corpus contains sequences from four organisms: yeast (Sac- 
charomyces cerevisiae, chromosomes 1, 4, 14 and the mitochondrial DNA), mouse (Mus muscu- 
lus, chromosomes 7, 11, 19, x and y), arabidopsis (Arabidopsis thaliana, chromosomes 1, 3 and 
4) and human (Homo sapiens, chromosomes 2, 13, 22, x and y). 

First, we present results that show the effectiveness of the proposed inverted repeats updating 
mechanism for finite-context modeling. Next, we show the advantages of using multiple (in 
this case, two) competing finite-context models for compression. 

4.1 Inverted repeats 

Regarding the inverted repeats updating mechanism, each of the sequences was encoded us- 
ing finite-context models with orders ranging from four to thirteen, with and without the 
inverted repeats updating mechanism. As in most of the other DNA encoding techniques, 
we also provided a fall back method that is used if the main method produces worse results. 
This is checked on a block by block basis, where each block is composed of one hundred DNA 
bases. As in the DNA3 version of Manzini's encoder, we used an order-3 finite-context model 
as fall back method (Manzini and Rastero, 2004). Note that, in our case, both the main and fall 
back methods rely on finite-context models. 

Table 4 presents the results of compressing the DNA sequences with the "normal" finite- 
context model (FCM) and with the model that takes into account the inverted repeats (FCM- 
IR). The bitrate and the order of the model that provided the best results are indicated. For 
comparison, we also included the results of the DNA3 compressor of Manzini and Rastero 
(2004). 

As can be seen from the results presented in Table 4, the bitrates obtained with the finite- 
context models using the updating mechanism for inverted repeats (FCM-IR) are always bet- 
ter than those obtained with the "normal" finite-context models (FCM). This confirms that the 
finite-context models can be modified according to the proposed scheme to exploit inverted 
repeats. Figure 5 shows how the finite-context models perform for various model orders, from 
order-4 to order-13, for the case of the "y-1" and "h-y" sequences. 

4.2 Competing finite-context models 

Each of the DNA sequences used by Manzini was encoded using two competing finite-context 
models with orders Mi, M2, 3 < Mi < 8 and 9 < M2 < 18. For each DNA sequence, the pair 
Mi,M2 leading to the lowest bitrate was chosen. The inverted repeats updating mechanism 
was used, as well as 5 = 1 for the lower-order model and 5 = 1/30 for the higher-order model. 
All information needed for correct decoding is included in the bit-stream and, therefore, the 
compression results presented in Table 5 take into account that information. The columns 
of Table 5 labeled "Mi" and "M2" represent the orders of the used models and the columns 
labeled with the percent sign show the percentage of use of each finite-context model. 
As can be seen from the results presented in Table 5, the method using two competing finite- 
context models always provides better results than the DNA3 compressor. This confirms that 
the finite-context models may be successfully used as the only coding method for DNA se- 
quences. Although we do not include here a comprehensive study of the impact of the 5 
parameter in the performance of the method, nevertheless we show an example to illustrate 
its influence on the compression results of the finite-context models. For example, using 5 = 1 
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Name 


Size 


DNA3 
bpb 


FCM 


FCM-IR 




Order 


bpb 


Order 


bpb 


y-1 


230203 


1.871 


10 


1.935 


11 


1.909 


y-4 


1 531 929 


1.881 


12 


1.920 


12 


1.910 


y-14 


784328 


1.926 


9 


1.945 


12 


1.938 


y-mit 


85 779 


1.523 


6 


1.494 


7 


1.479 


Average 


- 


1.882 


- 


1.915 


- 


1.904 


m-7 


5114647 


1.835 


11 


1.849 


12 


1.835 


m-11 


49909125 


1.790 


13 


1.794 


13 


1.778 


m-19 


703 729 


1.888 


10 


1.883 


10 


1.873 


m-x 


17430 763 


1.703 


12 


1.715 


13 


1.692 


m-y 


711 108 


1.707 


10 


1.794 


11 


1.741 


Average 


- 


1.772 


- 


1.780 


- 


1.762 


at-1 


29 830437 


1.844 


13 


1.887 


13 


1.878 


at-3 


23465336 


1.843 


13 


1.884 


13 


1.873 


at-4 


17550 033 


1.851 


13 


1.887 


13 


1.878 


Average 


- 


1.845 


- 


1.886 


- 


1.876 


h-2 


236268154 


1.790 


13 


1.748 


13 


1.734 


h-13 


95206001 


1.818 


13 


1.773 


13 


1.759 


h-22 


33 821688 


1.767 


12 


1.728 


12 


1.710 


h-x 


144 793 946 


1.732 


13 


1.689 


13 


1.666 


h-y 


22668225 


1.411 


13 


1.676 


13 


1.579 


Average 


- 


1.762 


- 


1.732 


- 


1.712 



Table 4. Compression values, in bits per base (bpb), for several DNA sequences. The "DNA3" 
column shows the results obtained by Manzini and Rastero (2004). Columns "FCM" and 
"FCM-IR" contain the results, respectively, of the "normal" finite-context models and of the 
finite-context models equipped with the inverted repeats updating mechanism. The order of 
the model that provided the best result is indicated under the columns labeled "Order". 



for both models would lead to bitrates of 1.869, 1.865 and 1.872, respectively for the "at-1", 
"at-3" and "at-4" sequences, i.e., approximately 2% worse than when using 8 = 1/30 for the 
higher-order model. 

Finally, it is interesting to note that the lower-order model is generally the one that is most 
frequently used along the sequence and also the one associated with the highest bitrates. In 
fact, the bitrates provided by the higher-order finite-context models suggest that these are 
chosen in regions where the entropy is low, whereas the lower-order models operate in the 
higher entropy regions. 



5. Conclusion 

Finite-context models have been used by most DNA compression algorithms as a secondary, 
fall back method. In this work, we have studied the potential of this statistical modeling 
paradigm as the main and only approach for DNA compression. Several aspects have been 
addressed, such as the inclusion of mechanisms for handling inverted repeats and the use 
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Fig. 5. Performance of the finite-context model as a function of the order of the model, with 
and without the updating mechanism for inverted repeats (IR), for sequences "y-1" and "h-y". 



of multiple finite-context models that compete for encoding the data. This study allowed us 
to conclude that DNA models relying only on Markovian principles can provide significant 
results, although not as expressive as those provided by methods such as MNL-1 or XM. Nev- 
ertheless, the experimental results show that the proposed approach can outperform methods 
of similar computational complexity, such as the DNA3 coding method (Manzini and Rastero, 
2004). 

One of the key advantages of DNA compression based on finite-context models is that the 
encoders are fast and have G(n) time complexity. In fact, most of the computing time needed 
by previous DNA compressors is spent on the task of finding exact or approximate repeats 
of sub-sequences or of their inverted complements. No doubt, this approach has proved to 
give good returns in terms of compression gains, but normally at the cost of long compression 
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Name 


Size 


DNA3 

bps 


FCM1 


FCM2 


FCM 




M 1 


% 


bps 


M 2 


% 


bps 


bps 


y-1 


230203 


1.871 


3 


82 


1.939 


12 


18 


1.462 


1.860 


y-4 


1 531 929 


1.881 


4 


88 


1.930 


14 


12 


1.470 


1.879 


y-14 


784328 


1.926 


3 


90 


1.938 


13 


10 


1.716 


1.923 


y-mit 


85 779 


1.523 


5 


83 


1.533 


9 


17 


1.178 


1.484 


Average 


- 


1.882 


- 


- 


1.920 


- 


- 


1.533 


1.877 


m-7 


5114647 


1.835 


6 


81 


1.907 


14 


19 


1.353 


1.811 


m-11 


49909125 


1.790 


4 


76 


1.917 


16 


24 


1.230 


1.758 


m-19 


703 729 


1.888 


4 


83 


1.920 


13 


17 


1.582 


1.870 


m-x 


17430 763 


1.703 


6 


70 


1.896 


15 


30 


1.081 


1.656 


m-y 


711 108 


1.707 


3 


66 


1.896 


13 


34 


1.199 


1.670 


Average 


- 


1.772 


- 


- 


1.911 


- 


- 


1.206 


1.738 


at-1 


29830437 


1.844 


6 


82 


1.898 


16 


18 


1.475 


1.831 


at-3 


23465336 


1.843 


6 


80 


1.901 


16 


20 


1.495 


1.826 


at-4 


17550 033 


1.851 


6 


80 


1.897 


15 


20 


1.560 


1.838 


Average 


- 


1.845 


- 


- 


1.899 


- 


- 


1.503 


1.831 


h-2 


236268154 


1.790 


4 


76 


1.905 


16 


24 


1.212 


1.755 


h-13 


95206001 


1.818 


5 


80 


1.895 


15 


20 


1.279 


1.723 


h-22 


33821688 


1.767 


3 


68 


1.925 


15 


32 


1.180 


1.696 


h-x 


144793946 


1.732 


5 


66 


1.901 


16 


34 


1.217 


1.686 


h-y 


22668225 


1.411 


4 


47 


1.901 


16 


53 


0.941 


1.397 


Average 


- 


1.762 


- 


- 


1.903 


- 


- 


1.212 


1.711 



Table 5. Compression values, in bits per symbol (bps), for several of DNA sequences. The 
"DNA3" column shows the results obtained by Manzini and Rastero (2004). Column "FCM" 
contains the results of the two combined finite-context models. The orders of the two models 
that provided the best result for each sequence are indicated under the columns labeled "Mi" 
and "M2". 



times. Although slow encoders could be tolerated for storage purposes (compression could 
be ran in batch mode), for interactive applications such as those involving the computation 
of complexity profiles (Dix et al., 2007) they are certainly not the most appropriate; faster 
methods, such as those examined in this chapter, could be particularly useful in those cases. 
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1. Introduction 

Intensive streams of video sequences arise more and more frequently in monitoring the qual- 
ity of production processes. Such streams not only have to be processed on-line, but also 
stored in order to document production quality and to investigate possible causes of insuf- 
ficient quality Direct storage of a video stream, coming with the intensity 10-30 frames per 
second with a resolution of 1-8 megapixels, from one production month would require 100-500 
terra bytes of a disk (or tape) space. A common remedy is to apply compression algorithms 
(like MPEG or H264), but compression algorithms usually introduce changes in gray-levels or 
colors, which is undesirable from the point of view of identifying defects and their causes. 
For these reasons we return to the traditional idea of sampling images, followed by loss-less 
compression. However, classical sampling on a rectangular grid is insufficient for our pur- 
poses, since it is still too demanding from the point of view of storage capacity. Our ex- 
perience of using equidistributed (or quasirandom) sequences as experimental sites in non- 
parametric regression function estimation Rafajlowicz and Schwabe (2003); Rafajlowicz and 
Schwabe (2006); Rafajlowicz and Skubalska-Rafajlowicz (2003) suggests that such sequences 
can be good candidates for sampling sites. Roughly speaking, the reason is in that the projec- 
tion of a 100 x 100 rectangular grid on the axes has 100 points, while a typical equidistributed 
sequence of the length 10 4 provides again 10 4 points when projected onto the same axes. The 
idea of using equidistributed (EQD) sequences in sampling images was firstly described in 
Thevenaz (2008), where it was used for image registration. Our goals are different and we 
need more specialized sampling schemes than a "general purpose" Halton's sequence, which 
was used in Thevenaz (2008). 

Our aim is to propose a new method of generating equidistributed sequences, which is based 
on space-filling curves. Due to the remarkable properties of space-filling curves (SFC), which 
preserve volumes and (to some extent) neighborhoods, the proposed sequences are well- 
suited for sampling of images in such a way that samples can be processed similarly as an 
original image. We concentrate mainly on 2D images here, but 3D images are also covered by 
the theoretical properties. Simple reconstruction schemes, which are well-suited for industrial 
images, are also briefly discussed. We also indicate ways of generating sampling sequences 
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and reconstructing underlying images by neural networks, which are based on weighted av- 
eraging of gray-levels of nearest neighbors. 

Let us note that space-filling curves have been used in image processing for image compres- 
sion Kamata et all (1996); Lempel and Ziv (1986); Schuster and Katsaggelos (1997); Skubal- 
ska-Rafajlowicz (2001b), dithering Zhang (1998); Zhang (1997) halftoning Zhang and Webber 
(1993) and median filtering Regazzoni and Teschioni (1997); Krzyzak (2001). However, the 
measure and neighborhoods-preserving properties of these curves were not fully exploited. 
The chapter is organized as follows. 

1. In Section 2 we collect some known and certain not so well-known properties of space- 
filling curves, including the Hilbert, the Peano and the Sierpinski curves. In addition to 
measure-preserving properties, we provide an efficient algorithms for calculating ap- 
proximations to selected space-filling curves. The definition and elementary properties 
of equidistributed sequences are recalled at the end of Section 2 with the emphasis on 
the Weyl sequences, which are used as the building block in the rest of the chapter. 

2. The proposed way of generating equidistributed sequences is presented in Section 3. It 
is based on transforming the Weyl one-dimensional sequence t[ = fractionalpart(i 9) , 
i = 1, 2, . . ., 6 - irrational, by a space-filling curve. We shall prove that sequences gen- 
erated in this way are also equidistributed. The choice of 6 is crucial for the practical 
behavior of the sampling scheme. Roughly speaking, should be an irrational number, 
which approximates badly by rational numbers. 

3. In Section 4 we discuss some properties of our equidistributed sequences as a sampling 
scheme for 2D images. 

• We shall prove that the spectrum of a wide class of images can be reconstructed 
from samples when their number grows to infinity. By "wide class" we mean 
measurable functions, which allow for discontinuities. 

• We exploit the measure-preserving properties of space-filling curves in order to 
show that moments of images can easily be approximated from samples. 

• It will also be shown how simple image processing tasks can be performed, utiliz- 
ing natural ordering of samples, which preserves neighbors in an image. 

4. In section 5 we discuss two algorithms for the approximate reconstruction of the under- 
lying image from samples. The first is based on the inversion of the spectrum estimate 
and it can be used for one image. The second one is based on the nearest neighbor (NN) 
technique, but it can be speeded up by preprocessing and storing (NN) addresses. This 
technique is useless for one image, but it is valuable when one needs to store a very 
long video sequence without degradation of pixel values, since NN addresses use only 
a very small portion of storage memory, while we gain on the reconstruction speed. 
The next reconstruction scheme, which is proposed here is based on neural networks of 
the radial-basis functions (RBF) type. We shall also provide the examples of sampling, 
processing and reconstructing industrial images. 

2. Preliminaries 

Our aim in this section is to collect known facts concerning space-filling curves and quasi- 
random sequences, which are useful for explaining the proposed way of sampling. 
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2.1 Space-filling curves - basic facts 

In the 19th and at the beginning of the 20th century, space-filling curves were developed and 
investigated as mathematical "monsters", since they are continuous, but nowhere differen- 
tiable. 

2.1.1 Definition 

From those pioneering times researches more frequently treat space-filling curves as useful 
tools. The first applications were in approximate, multidimensional integration, see, e.g., 
Kuipers and Niederreiter (1974). The next area where they happened to be useful is scan- 
ning images Lamarque and Robert (1996); Cohen et all (2007) and the bibliography cited 
therein. Note that scanning images by a space-filling curve is the task, which is different 
from our goals, since the curve is expected to visit all the pixels in an image. Thus, scan- 
ning along a space-filling curve provides only linear ordering of pixels. Furthermore, in the 
above-mentioned papers additional features of space-filling curves, such as their ability to 
preserve closeness or area, were not used. Scanning images with utilization of some proper- 
ties of space-filling curves for estimating the median was proposed in Krzyzak (2001). One 
more area of applications was proposed in Skubalska-Rafajlowicz (2001a), where space-filling 
curves were used as a tool in the Bayesian pattern recognition problems. 

Definition 1. A space-filling curve is a continuous mapping <J> : l\ °—f 1^, where Ij = [0, l] d is 
d-dimensional unit cube (or interval l\ = [0, 1]), d > 1. 

We cannot draw a space-filling curve, since it maps [0, 1] onto l^. Thus, the image of l\ by <J> 

would be completely black in the unit square. However, we can draw an approximation to 

such a curve, as is illustrated in Fig. 1. 

It is important to mention that these curves can be approximated to the desired accuracy by 

implementable algorithms (see below). 

The well-known curves constructed by Hilbert, Peano and Sierpihski possess properties 

Sagan (1994); Milne (1980); Moore (1900); Sierpihski (1912); Platzman and Bartholdi (1989); 

Skubalska-Rafajlowicz (2001a), which are stated in the two next subsections. These properties 

are stated for d = 2, but they holds for d > 2 with obvious changes. 

2.1.2 Most important properties 

The formula for changing variables in integrals, which is stated below, was used for con- 
structing multidimensional quadratures. Here, we shall need it for approximating the Fourier 
spectrum of images from samples. 

Property 1 (Fl - Change of variables). Let <J> : I\ — »• I^be a space-filling curve. Then, for every 
measurable function g : l^ —> R 

f g(x)dx= f 1 g(*(t))dt, (1) 

where x = [x^\x^] T and T denotes the transposition and the integrals in (1) are understood in the 
Lebesgue sense. 

The Lipschitz continuity of the curves constructed by Hilbert, Sierpihski and Peano is some- 
what more demanding property, than the continuity required in the above definition, but is 
less than necessary for the first order differentiability. 



134 



Signal Processing 




Fig. 1. An approximation to the Sierpinski SFC. 



Property 2 (F2 - Lipschitz continuity). There exists Q> > such that 

110(0-0(011 <c<s>\t-t'\ 1/2 , 

where I l.l I is the Euclidean norm in R 2 . 



(2) 



The Lipschitz continuity (2) is stated above for a 2D case and it reads intuitively as a distance 
preserving property in the sense that points close to each other in the interval are transformed 
by <J> onto points close together in l^, but the converse is not necessarily true, since curve O(f), 
t G h intersects itself many times. 
The next property will be useful for evaluating areas from samples along a space-filling curve. 

Property 3 (F3 - measure preservation). Space-filling curve <J> is the Lebesgue measure preserving 
in the sense that for every Borel A c h we have U2{A) = U\ (<J> -1 (A)), where u\ and u^ denote the 
Lebesgue measure in R\ and Ri, respectively. 

At first glance, this property is strange. Note that it means that only values of lengths and 
areas before and after the transformation by <J> are equal. For example, an interval of the 
length 0.1 cm is transformed into a set having the area 0.1 cm 1 . 

2.1.3 Quasi-inverses of space-filling curves 

As mentioned above, points which are close in li may have far, but not very far (see F2)) pre- 
images in l\. The reason is that does not have the inverse Sagan (1994) in the usual sense 
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(intuitively, because a curve intersects itself). For our purposes it is of interest to find at least 
one t G I\ such that 0(£) = x for given x. Consider a transformation Y : I2 —> l\, such that 
Y(x) G <E> -1 (x), where <E> _1 (x) denotes the inverse image of x, i.e., the set {t G I\ : 0(f ) = x}. 
<J> _1 allows to order linearly pixels in an image. We shall call Y a quasi-inverse of <J>. 

Property 4 (F4 - Quasi-invers). Let O : l\ — >> l&bea space-filling curve of the Hilbert, the Peano 
or the Sierpinski type. One can construct its quasi-inverse Y : Z^ — > l\ in such a way that it is also 
Lebesgue measure preserving. 

See Skubalska-Rafajlowicz (2004) for the constructive proof of this property. 

2.1.4 Remarks on generating space-filling curves 

It is important that there exist algorithms for calculating approximate value of the Peano, 

Hilbert and Sierpinski curves at a given point t G I\ with O ( |j of arithmetic operations, 
where e > denotes the accuracy of approximation Butz (1971); Skubalska-Rafajlowicz (2003); 
Skubalska-Rafajlowicz (2001a)). Furthermore, quasi-inverses of these curves can also be cal- 
culated with the same computational complexity Skubalska-Rafajlowicz (2004); Skubalska- 
Rafajlowicz (2001b); Skubalska-Rafajlowicz (2001a)). 

The specific self-similarities and the symmetries that space-filling curves usually possess, al- 
low us to define a given space-filling curve. For example, consider Sierpihski's 2D curve. 
O(f) = (x(t),y(t)) is uniquely defined by the following set of functional equations (see Sier- 
pinski (1912) for the equivalent definition) 

r x(t) = l/2-*(4t + l/2)/2, 

I y(*) = l/2-y(4* + l/2)/2 

[ 0<t < 1/8, 

f x(t) = l/2 + *(4(t-7/8))/2, 

{ y(*) = l/2-y(4(*-7/8))/2 

{ 7/8 < t < 1, 

r x(f) = l/2 + x(l-4(f-l/8))/2, w 

j y(0 = l/2-y(l-4(*-l/8))/2 

[ 1/8 < t<3/8, 

( x(t) = x(3/4-t) 

{ y(t) = l-y(3/4-0 

[ 3/8 < f < 7/8. 

It follows from (3) that x(0) = y(0) = and x(l/2) = y(l/2) = 1. After above observation, 
one can convert (3) into recurrent algorithm of computing O(f), £ G l\. If f has a finite bi- 
nary expansion, O(f) is obtained in a finite number of iterations. The code for generating the 
Sierpinski space-filling curve is provided in the Appendix. 

2.2 Equidistributed sequences in general 

Equidistributed sequences are deterministic sequences, which behave like random variables, 
which are drawn from a uniform distribution, but they are much more regular. They arise as 
a tool for numerical integration, which is applied like the well known Monte-Carlo method, 
but provides much more accurate results, at least for carefully selected sequences. 
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Definition 2. A deterministic sequence (xj)f =1 is called equidistributed (EQD) (or uniformly dis- 
tributed or quasi-random) sequence in Ij if 

n - 

}™ n ~ 1 Ls( x i)= L s( x ) dx ( 4 ) 

i=\ d 

holds for every continuous function g on 1^. 

We refer the reader to Kuipers and Niederreiter (1974) for account on properties of EQD se- 
quences and on their discrepancies, which are measures of their "uniformity". We shall use 
this definition mainly for d — \ and d = 2, but the properties, which are proved below hold 
also for d > 2. 
The well-known way of generating EQD sequences in [0, 1] is as follows 

^• = frac(z^), 1 = 1,2,..., (5) 

where the fractional part is denoted as frac(.), 6 is an irrational number. 

A large number of methods for generating multivariate EQD sequences have been proposed 
in the literature, including generalizations of (5), Van der Corput sequences, Halton sequences 
and many others Davis and Rabinowitz (1984); Kuipers and Niederreiter (1974). As far as we 
know, none of them have properties which are needed for our purposes. 

3. Generating sequences equidistributed along a space-filling curve 

We propose a new class of equidistributed multidimensional sequences, which is obtained 
from one-dimensional equidistributed sequences by transforming it by a space-filling curve. 
In fact, one can combine any reasonable way of generating a one-dimensional EQD sequence 
with one of the space-filling curves of the Hilbert, Peano or Sierpihski type. 

3.1 A new scheme of generating EQD sequences 

The proposed scheme of generating an equidistributed sequence along a space-filling curve is 
as follows. 

Step 1) Calculate £ z 's as in (5) (or as a one-dimensional Van der Corput sequence), 

Step 2) Select one of the above space-filling curves asO : I\ — >■ Ij and calculate x/s as fol- 
lows: 

x { = ®(ti),i= 1,2,..., n. (6) 

For given n and 6 it suffices to perform Steps 1) and 2) only once and store the resulting 
sequence x\, i — 1, 2, . . . , n. An example is shown in Fig. 2. 

Proposition 1. Sequence {x z }" =1 , Xj G R d , which is generated according to the above method is the 
equidistributed sequence in 1^. 

Proof. For continuous g : 1^ — » R, 

n- 1 t,g(xi)=n- 1 £>(<t>(f,)) -»• j\mt))dt= f g{x)dx, (7) 

since {f,}" = j are EQD, <£> is continuous, while the last equality follows from Fl).» 
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Fig. 2. The Sierpiriski SFC and n = 256 EQD points. 



3.2 Sampling of images 

Application of the above sequence for sampling images is straightforward, but requires some 
preparation. 

Preparation Perform Step 1 and Step 2, described in Section 3.1, for d = 2 in order to obtain 

EQD sequence [x- , x- ], i — 1, 2, . . . , n. 

Step 3 Scale and round sequence (6) as follows: 

n /z(0 = round(Nj l x- '), n v (i) = round(N v x- '), i = l,2,...,n, (8) 

where [n^(z'), n v (i)] denote coordinates of pixels in a real image, which has N^ pixels 
width and N v pixels height. 

Step 4 Read out samples fj = f([n%(i), n v (i)])), i = l,2,...,n. 

Remark 1. In practice, samples are collected as in Step 4 above, but for theoretical discussions we shall 
consider "theoretical" sample values ft = /(x/), i = 1, 2, . . . , n. 

Remark 2. Note that gray levels fi's are usually stored as integers from to 255, instead of [0, 1], as 
it is assumed about f and ft later on in this chapter. 



4. Properties of the sampling scheme 

This section is the central point of the chapter, since we collect here basic properties of the 
proposed sampling scheme. Some of them can be obtained by using known equdistributed 
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sequences, but properties presented in Section 4.3 and the reconstruction methods discussed 
in Section 5 essentially use unique features of sequences, which are equidistributed along a 
space-filling curve. 

4.1 Images as measurable functions 

Function / : R d — >- R is called measurable if for every c £ R the following level sets {x : 
f(x) < c} are measurable (see, e.g., Wheeden and Zygmund (1977) for the definition). In this 
chapter we treat images / as measurable functions. This is convenient from a mathematical 
point of view. On the other hand, the class of measurable functions is sufficiently wide to 
include real life gray level images. This class, in particular, contains discontinuous functions, 
which can be expressed as limits of sequences of continuous functions. Furthermore, limits 
of such limits are also measurable functions and this process can be iterated, leading again to 
measurable functions. 
Color images in RGB format can be modelled as triples of measurable functions. 

4.2 Spectrum approximation 

Denote by T(co), co = [co^\ co^] T the Fourier transform of image /, i.e., 

F(co) = [ exp(-jo; T x)f(x)dx, (9) 

Jl 2 

where j 2 = — 1. We approximate spectrum J 7 by 

T n (co) = n~ 1 f>xp(-jo; T x z -)^ (10) 

i=\ 

where x/s are EQD along a space-filling curve. 

Proposition 2. If f is measurable in 1^ and sampled at EDQ points along a space-filling curve, then 
for every co we have 

lim \T(co)-T n (co)\ =0. (11) 

The proof of this property is deferred to the Appendix. Note that this result was obtained 
without assuming that / is band-limited. For earlier results in this direction see Unser (1998). 
A detailed discussion of the convergence rate of T n (co) to T(co) is outside the scope of this 
chapter, since it requires some smoothness assumptions imposed on /. We only mention that 
if for / the Lipschitz condition with the exponent < oc < 1 holds, i.e., 

\f(x')-f(x")\<C f \\x'-x"\\«, 

where Cj > is a constant, then 

\T(co) - T n (co)\ <C (log(n) /nf n , 

where C > is a constant, which may depend on f , the kind of a space-filling curve and co, 
but not on n. 
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4.3 Fast approximate segmentation and blob analysis based on samples 

The segmentation of images is a basic technique for marking objects, which are characterized 
by (approximately) the same gray level. In other words, our aim is to mark (approximately) 
regions such that 

{x e h : G 1 < f(x) < G 2 }, (12) 

where < G\ < G 2 < 1 are specified thresholds. The next step is to find blobs, which are 
cluster of points, which are close to each other and far from points, which belong to another 
cluster. The segmentation and blob analysis task is time consuming, since it requires to not 
only visit each pixel and to mark it (as black, say), if G\ < f(x) < G 2 , but also to group 
marked into clusters, which usually requires to visit marked pixels several times (see, e.g., 
Davies (2005)). 

We can reduce the computational burden by performing the segmentation and the blob anal- 
ysis directly on samples. The blob analysis is also called silhouettes analysis, which are ex- 
tracted by the segmentation. 

Assume that sample points (ti, fi), i = 1, 2, . . . , n are reordered according to their first co- 
ordinates. Denote by tn\ z'-th point of the equidistributed sequence in I\. Thus, after sorting 
t({\ < t(i-\-i)f i = 1, 2,. . . , n. Simultaneously - we keep the corresponding gray levels, which 
are denoted by fn\ 's, i.e., 

/(,•)=/(*(*(,•))), » = l,2,...,n (13) 

Thus, our samples have the form {tn\, /(/))• Now the procedure for approximate segmentation 
and blob analysis runs as follows. 

Segmentation For each sample point mark tn\ as "black", if 

G 1 </ (0 <G 2 f = l,2,...,n. 
Blob analysis Starting from tny search for the first group of consecutive points 

Hv) <f (p+i) < ••• <*(*)' 1 ^p<i> 

which are marked as "black". Then, repeat this search starting from t( q +2) (^(a+i) cannot 
be a member of the first group) and find the second group etc. Attach a label, e.g., 
number or color, to each group and treat it as the approximation of a blob. 

Measuring blobs For each blob calculate the difference between the last point and the first 
point, i.e., tf q \ — tr p \ and treat it as the approximation of the area of the corresponding 
blob. 

The segmentation step does not require explanation (see Fig. 3). In the second step we use 
F2) property of space-filling curves that is if points tn\ and tn+\\ are close, the also points 
xn\ = <f>(ttj\) and *(/+i) = ^(fy+i)) are close in the image. To justify the last step, let us note 

that, according to F3) and F4), the length \tr q \ — tr p \ can be used as the approximation of the 

area of the smallest polygon containing xu\, ) = p, p + 1, . . . , q. 

The idea of the approximate blob analysis is illustrated in Fig. 3. The white, gray and black 
squares (left panel) were sampled in 512 points, which are equidistributed along the Sierpihski 
space-filling curve. The resulting gray levels are shown in the right panel. Note that samples 
from the white square are almost perfectly grouped as samples, which are numbered as 320 to 
440. Similarly, samples from the black and light gray squares are grouped in the right panel as 
samples from 60 to (almost) 200 and from 200 to 320, respectively. Samples from the dark gray 
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square are split into two groups. The first one is numbered from 1 to 60. the second one, from 
420 to 512. The consequence of this (unavoidable) split is not too severe, since we obtain two 
blobs of dark gray color (if G\ « 185, G^ ~ 195) instead of one, but when transformed to the 
image space, these two blobs will be close to each other. The only points, which would lead to 
false grouping are shown as separate points in the right panel of Fig. 3. This is the price paid 
for speeding up grouping. We can avoid even these false classifications by checking a proper 
classification of small clusters, but at the expense of an additional computational burden. The 
above approach can be applied to images in RGB format, just by applying it to each channel 
separately, but keeping the same sequence tn\'s. 





Fig. 3. Explanation why (approximate) segmentation and blob analysis work. 



4.4 Approximating moments 

Moments % of image / with respect to linearly independent functions v^(x) are defined as 



a k — / f(x)v] c (x)dx, k = l,2,. 

J h 



(14) 



flfc's are usually approximated by the sums of gray levels located at all the pixels. We can gain 



much on efficiency using a. 






1 Ef=i fi to evaluate theoretical moments from samples. 



Proposition 3. Let f and v^,k = 1, 2, . . . be measurable functions in li. Then, for fi sampled at 
joints X(, which are equidistributed along a space-filling curve, we have 



lim loci, — oc) ' 



0, 



(15) 



i.e., approximate moments converge to the theoretical moments as the number of samples grows to 
infinity. 

We omit the proof, since it is similar to the one for the spectrum approximation. 
The role of moments in image analysis is well established (see, e.g., Davies (2005); Pawlak 
(2006). In particular, selecting vj c (x) / s as ordered monomials one can evaluate centroids of 
blobs, their area etc. Central moments, in turn, provides translation invariant information 
about shape parameters describing blobs. 
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4.5 Moving mean and median filtering 

The moving mean and the moving median are the most popular filters applied in image pro- 
cessing. A rectangular window of size (2P + 1) x (2 Q + 1), say, is scanning the image and 
the mean value (or the empirical median) of gray levels of the corresponding pixels replaces 
the central pixel value. 

Assuming that samples are ordered as in Section 4.3, one can perform (approximately) the 
same kind of filtering directly on samples. The filtering process runs as follows. 

Step 1 Sort samples according to their first coordinates in order to obtain [tny /m )• 

Step 2 Select half of the size of a neighborhood, which is used for filtering. Denote it by S. 

Step 3 Starting from i = S + 1 to i = n — (S + 1), perform the following operations: 

1. calculate 

f i = (2S + l)- 1 £ f (i _ m) (16) 

m= — S 
or the empirical median of the following gray levels 

{/(,-») : « = -s 0,1 S}, 

2. fi (or by the empirical median) is attached to the point tn\ in the filtered sample. 

As usual, we are faced with the boundary problem, since we cannot filter samples numbered 
by i < S and i > n — S. The simplest remedy is to leave these samples unchanged. 
Somewhat more sophisticated way of median filtering along a space-filling curve was pro- 
posed in Krzyzak (2001), but - in opposite to the present chapter - neighbors were not equidis- 
tributed. 

As is known, sampling of images and sapce-filling curves have many other applications (see, 
e.g., Davies (2001); Lamarque and Robert (1996)), in which the sampling scheme proposed 
here can also be useful. 

5. Approximate reconstruction by k-nearest neighbors RBF nets 

Our aim is to demonstrate that images can be efficiently reconstructed from the samples, 
which are equidistributed along a space-filling curve. We shall concentrate on reconstruc- 
tion schemes, which are based on nearest neighbors and artificial neural networks from the 
radial basis functions (RBF) class. 

An alternative way would be to estimate the spectrum of an image according to (10) on a 
regular grid and to calculate the inverse discrete Fourier transform by the FFT algorithm. 

5.1 Reconstruction using RBF nets and exact neighbors 

Consider N^ x N v image. The coordinates of its pixels are denoted as (h, v), while positions 
of sample points are denoted as (n^(z'), n v (i)), i = 1, 2, . . . , n. Abusing the notation, we shall 
write ff^m)' forgetting for a while that earlier / was defined in [0, l] 2 . 
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5.1.1 1-NN reconstruction scheme 

A seemingly naive algorithm of reconstructing the underlying image is the following. 

Preparations For all N^ N v positions (h, v) pixels find the nearest neighbor (1-NN) among 
positions of samples (n^(z'), n v (i)), i = 1, 2, . . . , n and store these positions in N^ x N v 
table C, say Its elements c(h, v), h = 1, 2 . . . , N^, v = 1, 2 . . . , N v contain addresses to 
the closest sample point. 

Step 1 Repeat Step 2 for h = 1, 2 . . . , N h , v = 1, 2 . . . , N v . 

Step 3 Attach gray level of the nearest sample point f( c (h fV )) to pixel (h, v). 

The most time consuming Step 1 is performed only once. As a result we obtain table C, which 
is of the same size as an original image and - at a first glance - all the compression effect is 
distracted. Note however, that when frames from a long video sequence are sampled and later 
some of them have to be reconstructed, then it pays to store matrix C in order to have an al- 
most immediate reconstruction of selected frames. This is exactly the case when a production 
quality is monitored by a camera and we have to store (and keep for a long time) very long 
sequences of images, which document the quality of products. 

5.1.2 k-NN reconstruction using RBF net 

We can generalize the above reconstruction scheme by taking into account gray levels of near- 
est neighbors starting from the first one, second nearest up to k-th nearest. It is convenient to 
express such a generalized reconstruction scheme as a neural network from the well known 
radial basis functions (RBF) class. 
To this end, we select a nonnegative kernel K : Ri —> Ri, which is a function such that 

/OO rCO 

tK(t)dt = 0, / t 2 K(t)dt < oo, (17) 

-00 J — oo 

which is normalized K(0) = 1. This kind of normalization is not typical, but convenient 
for our purposes. Typical examples include the uniform kernel (K(t) = 1, \t\ < 1 and zero 
otherwise), the Epanechnikov kernel etc. 
Denote by f(h,v) * ne reconstructed gray level at (h, v), which is calculated as follows 

k 
f(h, v) = Ys W i( h > V ) fc(j, h, v) > ( 18 ) 

where c(j, h, v) is the address of ;-th closest point among positions of samples (%(i), n v (i)), 
i = 1, 2, . . . , n, while weights Wj(h, v) are defined as follows: 

= Km.*)-c(!.Kv)\mw) t (19) 

; I$ =1 K(\\(h,v)-c(j,h,v)\\yH(k)) 

where 

H(k) d = \\(K v)-c(k,h, v)\\ 2 . (20) 

Note that when k = 1 and kernel K is the uniform one, then (18) reduces to 1-NN reconstruc- 
tion scheme. 

We remark that (18) is the approximation scheme rather than interpolatory one, as it was used 
in Anton et all (2001). 
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5.2 Reconstruction using RBF nets and neighbors along SFC 

We can reduce the computational burden on finding nearest neighbors by replacing the exact 
search by the approximate one, which is performed along a space-filling curve. The proposed 
method is as follows. 

1-NN along SFC 

Step 1 For all pixels (h, v) perform the following steps: 

def 

1. normalize current pixel (h, v) to l^ as x^ /V = (h/N^, v/N v ). 

def 

2. calculate its quasi-inverse fy v = Y(x/ Zi; ) 

3. find its nearest neighbor among all tn\'s and denote its number by c(h, v). 

and store the resulting N^ x N v matrix as C. 
Step 2 As the approximate value of / at pixel (h, v) (or at x^ v ) take / at c(h, v). 
Step 3 Repeat Step 2 for all (h, v). 

The main advantage of this scheme is in that finding NN among ordered tn\'s has computa- 
tional complexity 0(log 2 (n)). The price for that is a possibility of missing the true NN in li, 
since in Step 1 we use the quasi-inverse of SFC. Nevertheless, a point found in this is close to 
NN in I2 due to F2). Matrix C can be treated as approximation of matrix C in the sense that 
many of its entries are the same as the corresponding entries of matrix C. The differences arise 
due to self-crossing of SFC. 

We do not provide details of reconstruction by RBF net, which is based on approximate nearest 
neighbors, since changes in (19) and (20) are obvious. 

5.3 Reconstruction by local random spreading of grey levels 

In opposite to the above-described reconstruction schemes, which are based on searching (ap- 
proximate) neighbors to each pixel, the method considered here spreads gray levels of sam- 
ples in their neighborhoods. Below, we describe the simplest way of such spreading, which is 
based on a random choice of neighbors. 

Reconstruction by random spreading 

Step 1 Prepare N^ x N v matrix S, say, as follows. Fill its entries, denoted as s(h, v) by sam- 
pled gray levels at appropriate positions. The remaining entries fill by "empty" symbol 
(coded as a number greater than 1 (or 255)). 

Step 2 Check whether "empty" entries are present in S. If not, the stop and S contains the 
reconstructed image. Otherwise, go to Step 3. 

3 Find the position of the next "empty" element of matrix S and denote it by (/z, v). 

Step 4 Select at random (with equal probabilities) one of the following directions "up", 
"down", "left", "right". 

Step 5 Assign the contents of s(h — 1, v) to s(h, v), if the direction is "left". Assign the contents 
of s(h + 1, v) to s(h, v), if the direction is "right" etc. Go to Step 2. 

In Step 5 it may happen that the contents assigned to s(h, v) is still "empty", but after a short 
time gray levels of samples nicely "smear" over the image. The result of the reconstruction 
is random, but repeated reconstructions produce visually stable images in a relatively short 
time. In Steps 4 and 5 one can use the neighborhood containing eight or more pixels. 
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5.4 Examples 

As explained in the Introduction, the proposed sampling and reconstruction schemes are ded- 
icated mainly for industrial images. However, it is instructive to verify their performance us- 
ing the well-known example, which is shown in Fig. 4. Analysis of the differences between 
the original and the reconstructed images indicate that 1-NN reconstruction scheme provides 
the most exact reconstruction, but the reconstruction by random spreading provides the nicest 
looking image. 

The application to industrial images is illustrated in Fig. 5, in which a copper slab with defects 
is shown. Note that it suffices to store 4096 samples in order to reconstruct 1000 x 1000 image, 
without distorting gray levels of samples from the original image. This is equivalent to the 
compression ratio of about 1/250. Such a compression rate plus loss-less compression allows 
us to store a video sequence (30 fps) from one month of a continuous production process on a 
disk or tape, having 1 TB (terra byte) capacity. 



6. Appendix - proof of Proposition 3 

Take arbitrary e > 0. By the Lusin theorem, there exists a set £ = E(e/4) such that /|e is 
continuous and ]It\E — h) < € /^- Denote by J-'e(co) the Fourier transform of /|jf. Then, for 

def 
D = E — I2 we have 



\T{w)-T E {w) 
since both integrands do not exceed 1. Let 



/o< 



-jar x 



f(x) dx 



< niP) < i> 



T E (cv) = n l ^ exp(-}cv T Xi)fi. 

x t eE 



Define A n = \T n (cv) — ^e(<^) I- Then 



A n 



Clearly, A n < Af(I 2 - E)/n, where 



n 1 E ex P(~) ^^fi 






N(h ~ E) = card{z : x { e (I 2 - E)}. 
From Proposition 1 it follows that for n —> 00 

M{1 2 - £) 



A„< 



Mh ~ E) < e/4. 



Thus, for n sufficiently large we have A n < e/4. Define 



\-jm)^ E eM ~ ]u,Txi)fi 






where J\f(E) = card{i : x t eE}. Clearly, 



E ex P(-)U T Xi)fi 

Xi€E 



<Af(E) 



(21) 



(22) 



(23) 



(24) 
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Fig. 4. Lena image, 512 x 512 pixels, (upper-left panel) sampled at 10 000 points equidis- 
tributed along the Sierpinski space-filling curve (upper-middle panel). Gray levels at sample 
points are shown in the upper-right panel. The results of reconstruction by 1-NN method 
(middle left panel), by 1-NN along the space-filling curve (central panel) and by spread to 
random-NN (middle right panel). The differences between the original image and the recon- 
structed one are shown in the last row of this figure. 



Thus, for n large enough 

S n < \(Af(E)/n-l\)<e/4, (25) 

since, by Proposition 1, \(Af(E)/n — }t2{h)\) -> as n ^ oo. We omit argument to in the 
formulas that follow. Summarizing, we obtain. 



\T-T n \ <e/4+\F E - f n \, 



(26) 
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Fig. 5. Copper slab with defects, 1000 x 1000 pixels (upper left panel) and its reconstruction 
from n — 2048 samples by 1-NN method (upper right panel). The same slab reconstructed 
from n = 4096 samples (lower left panel) and the difference between the original image and 
the reconstructed one (lower right panel). Compression ratio 1/250. 



since, by (21), \T — Te\ < e/A. Analogously, 

\T E - T n \ <e/4+ \T E -T n \ 
due to (24). Finally, 



+ 



?E 






(27) 
(28) 



The last term in (28) approaches zero, since / is continuous in E and Proposition 1 holds. 
Hence, \Te — Fe\ < e /2 for n large enough, due to (25). Using this inequality in (27) and 
invoking (26) we obtain that for n large enough we have | T — T n | < e. • 
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7. Appendix - Generating the Sierpihski space-filling curve and equidistributed 
points along it. 

In this Appendix we provide implementations of procedures for generating points from the 
Sierpihski space-filling curve and its quasi-inverse, which are written in Wolfram's Mathe- 
matica language. Special features of new versions of Mathematica are not implemented with 
the hope that the code should run and be useful for all versions, starting from version 3. 
The following procedure tranr calculates one point of the Sierpihski curve, i.e., for given 
t G I\ an approximation to 0(t) G Id is provided, but only for d > 2 and even. Parameter 
k of this procedure controls the accuracy to which the curve is approximated. It should be a 
positive integer. In the examples presented in this chapter k = 32 was used. 

tranr [d_, k_, t_] := Module [ {bd, cd, ii, j , j j, tt, KM, km, be, kb } , 

bd=l; tt :=t;xx={l}; 

Do [bd=2 A ii-bd+l; AppendTo [xx, 1 ] , { ii, d-1 } ] ; 

cd=bd*2 A (-d) ; km={ } ; 

Do [kb=Floor [ (tt-cd/2 A d) *2 A d] +1; 

tt=2 A d* (tt-cd/2 A d- (kb-1) *2 A (-d) ) ; 
If[kb==2 A d, kb=0]; 

If[ Floor [kb/2] <kb/2,tt=l-tt] ; AppendTo [km, kb] , {j,k}]; 
Do[ KM=km[ [k-j+1] ] ; ww={}; 

Do[ If[KM< 2 A (d-j j) ,be=0,be=l] ; AppendTo [ww, be] ; 
KM=KM-be*2 A (d-j j) ; 

If [be==l,KM=2 A (d-jj)-KM-l] , { jj,d}]; 
Do[xx[[d-jj+l]]=l/2-(l/2-ww[[jj]])*xx[[d-jj+l]] , {jj,d}] , {j,k}]; 
(*out*) xx] 

The following lines of the Mathematica code generate the sequence of 2D points, which are 
equidistributed along the Siepinski space-filling curve. 

dim = 2; deep = 32; n = 512; th = (Sqrt[5.] - 1 . ) /2 . ; {i, 1, n}]]; 

points = Map [tranr [dim, deep, #] &, Sort [Table [FractionalPart [i*th] ] ; 
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1. Introduction 



Periodicities are found in speech signals, musical rhythms, biomedical signals and machine 
vibrations. In many signal processing applications, signals are assumed to be periodic or 
quasi-periodic. Especially in acoustic signal processing, signal models based on periodicities 
have been studied for speech and audio processing. 

The sinusoidal modelling has been proposed to transform an acoustic signal to a sum of 
sinusoids [1]. In this model, the frequencies of the sinusoids are often assumed to be 
harmonically related. The fundamental frequency of the set of sinusoids has to be specified 
for this model. In order to compose an accurate model of an acoustic signal, the noise-robust 
and accurate fundamental frequency estimation techniques are required. Many fundamental 
frequency estimation techniques are performed in the short-time Fourier transform (STFT) 
spectrum by peak-picking and clustering of harmonic components [2] [3] [4]. These 
approaches depend on the frequency spectrum of the signal. 

The signal modeling in the time-domain has been also proposed to extract a waveform of an 
acoustic signal and its parameters of the amplitude and frequency variations [5]. This 
approach aims to represent an acoustic signal that has single fundamental frequency. For 
detection and estimation of more than one periodic signal hidden in a signal mixture, 
several signal decomposition that are capable of decomposing a signal into a set of periodic 
subsignals have been proposed. 

In Ref. [7], an orthogonal decomposition method based on periodicity has been proposed. 
This technique achieves the decomposition of a signal into periodic subsignals that are 
orthogonal to each other. The periodicity transform [8] decomposes a signal by projecting it 
onto a set of periodic subspaces. In this method, seeking periodic subspaces and rejecting 
found periodic subsignals from the observed signal are performed iteratively. For reduction 
of the redundancy of the periodic representation, a penalty of sparsity has been introduced 
to the decomposition in Ref. [9]. 

In these periodic decomposition methods, the amplitude of each periodic signal in the 
mixture is assumed to be constant. Hence, it is difficult to obtain the significant 
decomposition results for the mixtures of quasi-periodic signals with time-varying 
amplitude. In this chapter, we introduce a model for periodic signals with time-varying 
amplitude into the periodic decomposition [10]. In order to reduce the number of resultant 
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periodic subsignals obtained by the decomposition and represent the mixture with only 
significant periodic subsignals, we impose a sparsity penalty on the decomposition. This 
penalty is defined as the sum of h norms of the resultant periodic subsignals to find the 
shortest path to the approximation of the mixture. The waveforms and amplitude of the 
hidden periodic signals are iteratively estimated with the penalty of sparsity. The proposed 
periodic decomposition can be interpreted as a sparse coding [15] [16] with non-negativity 
of the amplitude and the periodic structure of signals. 

In our approach, the decomposition results are associated with the fundamental frequencies 
of the source signals in the mixture. So, the pitches of the source signals can be detected 
from the mixtures by the proposed decomposition. 

First, we explain the definition of the model for the periodic signals. Then, the cost function 
that is a sum of the approximation error and the sparsity penalty is defined for the periodic 
decomposition. A relaxation algorithm [9] [10] [18] for the sparse periodic decomposition is 
also explained. The source estimation capability of our decomposition method is 
demonstrated by several examples of the decomposition of synthetic periodic signal 
mixtures. Next, we apply the proposed decomposition to speech mixtures and demonstrate 
the speech separation. In this experiment, the ideal separation performance of the proposed 
decomposition is compared with the separation method obtained by an ideal binary 
masking [10] of a STFT. Finally, we provide the results of the single-channel speech 
separation with simple assignment technique to demonstrate the possibility of the proposed 
decomposition. 

2. Periodic decomposition of signals 

For signal analysis, the periodic decomposition methods that decompose a signal into a sum 
of periodic signals have been proposed. Most fundamental periodic signal is a sinusoid. In 
speech processing area, the sinusoidal modeling [1] that represents the signal into the linear 
combination of sinusoids with various frequencies is utilized. The sinusoidal representation 
of the signal /(n) with constant amplitude and constant frequencies is obtained as the form of 

f{n) = Y j A j cos(cD j n + ^). (1) 

7=1 

This model relies on the estimation of the parameters of the model. Many estimation 
techniques have been proposed for the parameters. If the frequencies {coj}i<j<j are 
harmonically related, all frequencies are assumed to be the multiples of the fundamental 
frequency. To detect the fundamental frequencies from mixtures of source signals that has 
periodical nature, multiple pitch detection algorithms have been proposed [2] [3] [4]. 
The signal modelling with (1) is a parametric modeling of the signal. On the contrast, the 
non-parametric modeling techniques that obtain a set of periodic signals that are specified in 
time-domain have been also proposed. 

For time-domain approach of the periodic decomposition, the periodic signal is defined as a 
sum of time-translated waveforms. Let us suppose that a sequence {f p (n)}o<n<N is a finite 
length periodic signal with a length N and an integer period p>2. It satisfies the periodicity 
condition with an integer period p and is represented as 

/»=<v(»)2»-*p) a) 
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where K = \_(N-l)/p] that is the largest integer less than or equal to (N-l)/p. The sequence 
{tp(n)}o< n <p corresponds to a waveform of the signal within a period and is defined over the 
interval [0, p-1]. t p (n) = for n > p and n < 0. This sequence is referred to as the p-periodic 
template. The sequence {a(n)}o< n <N represents the envelope of the periodic signal. If the 
amplitude coefficient a(n) is constant, the model is reduced to 

/»=£*„(n-*p). (2) 

k=0 

Several periodic decomposition methods based on the periodic signal model (2) have been 
proposed [6] [7] [8] [9]. These methods decompose a signal i(n) into a set of the periodic 
signals as: 

/(»)=ii»- fc p) ( 3 ) 

where P is a set of periods for the decomposition. This signal decomposition can be 
represented in the matrix form as: 

f = ZU p t p (4) 

peP 

where t p is the vector which corresponds to the p-periodic template. The z-th column vector 
of A p represent an impulse train with a period p. The elements of U p are defined as 

f 1 for n = kp + i - 1 where A: = 0, 1, ■ • • 

Un ' l = {0 otherwise ' (5) 

The subspace that is spanned by the column vectors of U p is referred to as the p-periodic 
subspace [8] [9]. 

If the estimations of the periods hidden in signal f are available, we can choose the periodic 
subspaces with the periods that are estimated before the decomposition. For MAS [6], the 
signal is decomposed into periodic subsignals as the least-squares solution along with an 
additional constrained matrix. In Ref. [8], the periodic bases are chosen to decompose a 
signal into orthogonal periodic subsignals. Therefore, these methods require that the 
number of the periodic signals and their periods have to be estimated before decomposition. 
Periodic decomposition methods that do not require predetermined periods have also been 
proposed. In Ref. [7], the concept of periodicity transform is proposed. Periodicity transform 
decomposes a signal by projecting it onto a set of periodic subspaces. Each subspace consists 
of all possible periodic signals with a specific period. In this method, seeking periodic 
subspaces and rejecting found periodic subsignals from an input signal are performed 
iteratively. Since a set of the periodic subspaces lacks orthogonality and is redundant for 
signal representation, the decomposition result depends on the order of the subspaces onto 
which the signals are projected. In Ref. [7], four different signal decomposition methods - 
small to large, best correlation, M-best, and best frequency - have been proposed. In Ref. [9], 
the penalty of sparsity is imposed on the decomposition results in order to reduce the 
redundancy of the decomposition. 

In this chapter, we discuss the decomposition of mixtures of the periodic signals with time- 
varying amplitude that can be represented in the form of (1). To simplify the periodic signal 
model, we assume that the amplitude of the periodic signal varies slowly and can be 
approximated to be constant within a period. By this simplification, we define an 
approximate model for the periodic signals with time-varying amplitude as 
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/ P M=2X^( n_/c p)- ( 6 ) 

k=0 

In order to represent a periodic component without a DC component, the average of f p (n) 

over the interval [0, p-1] is zero. The amplitude coefficients a p ,k are restricted to non-negative 

values. 

These p-periodic signals can also be represented in a matrix form as well as the previous 

periodic signal model. The matrix representation of (6) is defined as 

f p =A p t,, (7) 

In this form, the amplitude coefficients and the template are represented in an N by p matrix 
A p and a /^-dimensional template vector t p , which is associated with the sequence t p (n), 
respectively. A p is a union of the matrices as 

A p ={D pA ,D p , 2 -D p , K+1 } (8) 

where superscript T denotes transposition. 

{ ^D P/ j}i<j<k+i are p by p diagonal matrices whose elements correspond to a Pr j.\. T) Pi k+i is the p 
by N-pK matrix whose non-zero coefficients that correspond to a Pi k appear only in (z, z) 
elements. Since only one element is non-zero in any row of the A p , the column vectors of Ap 
are orthogonal to each other. The 12 norm of each column vector is supposed to be 
normalized to unity. In (6), the average of the waveform over the interval [0, p-1] must be 
zero. Hence, the condition 

«X=0 (9) 

where u p is a vector, of which elements correspond to the diagonal elements of Y> Pi \. 
Alternatively, the p-periodic signal in (2) can be represented as 

f p =T p a p . (10) 

In this form, the amplitude coefficients and the template are represented in a N by K+l 
matrix T p and K+l -dimensional amplitude coefficients vector a p whose elements are 
associated with the amplitude coefficients a Pl k, respectively. T p consists of the column 
vectors that correspond to the shifted versions of the p-periodic template. As same as A p , 
only one element is non-zero in any row of T p . So, we defined Tp as the matrix which 
consists of the normalized vectors that are orthogonal to each other. 

In this study, we propose an approximate decomposition method that obtains a 
representation of a given signal f as a form: 

f = e+Z f P (11) 

where e is an approximation error between the model and the signal f . 

We suppose that the signal f is a mixture of some periodic signals that can be approximated 
by the form of (2), however, the periods of the source signals are unknown. So, we specify 
the set of periods P as a set of all possible periods of the source signals for the 
decomposition. If the number of the periods in P is large, the set of the periodic signals 
{fp }p e P that approximate the signal f with small error is not unique. To achieve the 
significant decomposition with the periodic signals that are represented in the form of (2), 
we introduce the penalty of the sparsity into the decomposition. 
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3. Sparse decomposition of signals 

In Ref. [15] [16] [17], sparse decomposition methods that are capableof decomposing a signal 
into a small number of basis vectors that belong to an overcomplete dictionary have been 
proposed. Basis pursuit (BP) [17] is a well known sparse decomposition method and 
decomposes a signal into the vectors of a predetermined overcomplete dictionary. The 
signal f is represented as Oc, where O and c are the matrix that contains the normalized 
basis vectors and the coefficient vector, respectively. 

In sparse decomposition, the number of basis vectors in O is larger than the dimensionality 
of the signal vector f. For this decomposition, the penalty of the sparsity is defined as l\- 
norm of c. The signal decomposition by BP is represented as a constrained minimization 
problem as follows: 

minlc^ subject to f = Oc (12) 

where ||-| denotes the h norm of a vector. 

Since the /i-norm is defined as the sum of the absolutes of the elements in the coefficient 

vector c, BP determines the shortest path to the signal from the origin through the basis 

vectors. The number of the basis vectors with nonzero coefficients obtained by choosing the 

shortest path is much smaller than the least square solution obtained by minimizing the h- 

norm [17]. 

Usually, (12) is solved by linear programming [17]. However, it is difficult to apply linear 

programming to the large number of samples that appear in signal processing applications. 

So, an approximation of the solution of BP is obtained from the penalty problem of (12) as 

follows: 

i 

c = argmin — |f-Oc|| 2 +l||c| 1 (13) 

where X denotes a Lagrange multiplier. ||-|| denotes the h norm of the vector. This 

unconstrained minimization problem is referred to as a basis pursuit denoising (BPDN) [17] 
[18]. When O is specified as a union of orthonormal bases, an efficient relaxation algorithm 
can be applied [18]. 

From Bayesian point of view, the minimization (13) is the equivalent of MAP estimation of 
the coefficient vector c under the assumption that the probability distribution of each 
element of the coefficient vector is an identical Laplace distribution [15]. 
The dictionary O is fixed for signal representation in the BP and BPDN. In a sparse coding 
strategy [15] [16], the dictionary O is adapted to the set of the signals. The dictionary is 
updated with the most probable one under the estimated sparse coefficients and the set of 
the signals [15]. 

For our periodic decomposition, we also impose the sparsity penalty on the decomposition 
under the assumption that the mixture contains a small number of periodic signals that can 
be approximated in the form of (6). Our objective is to achieve signal decomposition to 
obtain a small number of periodic subsignals rather than basis vectors. In order to achieve 
this, we define the sparsity measure as the sum of h norms of the periodic subsignals to find 
the shortest path to the approximation of the signal as well as BPDN. 
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4. Sparse periodic decomposition 

4. 1 Cost function for periodic decomposition 

For our periodic decomposition, we also impose the sparsity penalty on the decomposition 
under the assumption that the mixture consists of a small number of periodic signals that 
can be approximated in the form of (2). Our objective is to achieve signal decomposition 
with a small number of periodic subsignals rather than the basis vectors. In order to achieve 
this, the probability distribution of the h norm of each periodic signal is assumed to be a 
Laplace distribution, and then the probability distribution of the set of the periodic signals is 

jffeUVn^UlM)- ^ 

The noise is assumed to be Gaussian, and then the conditional probability distribution of f is 



Pi 



fcL 



:exp 



' 2/1 



'-2X 



peP 



2 N \ 



(15) 



Along with Bayes' rule, the conditional probability distribution of the set of the periodic 
signals is 



p (fcL lf 



{trlMrL 



(16) 



Substituting the prior distributions of the periodic signals and the noise into (16), we can 
derive the likelihood function of the set of periodic signals. From the likelihood function, we 
define the cost function E for the periodic decomposition as: 



-IX 

peP 



peP 



"pfph 



(17) 



In our periodic decomposition, a signal f is decomposed into a set of periodic subsignals 
while reducing the cost E and maximizing the likelihood. 

In the cost for BPDN (12), the sparsity penalty is defined as the Zi-norm of the coefficient 
vector that is identical the total length of the decomposed vector of the signal. In our 
periodic decomposition, the sparsity penalty is also defined as the sum of the decomposed 
vectors that are represented in the form of the periodic signal model shown in (6). 

4. 2 Algorithm for sparse periodic decomposition 

To find the set of the periodic subsignals {f p } pe p, we employ a relaxation algorithm. This 
relaxation algorithm always updates one chosen periodic subsignal while decreasing the 
cost function (17). The template vector t p and amplitude vector a p of the chosen period p are 
alternatively updated in an iteration. In the algorithm, we suppose that the set of the periods 
P consists of M periods which are indexed as {pi . . ./?m}- 
The relaxation algorithm for the sparse periodic decomposition is as follows: 



1) Set the initial amplitude coefficients for {A p } 

2) z = l 

3) Compute the residual 



Sparse signal decomposition for periodic signal mixtures 



157 



= f -Z f P 



(18) 



4) Represent f as At . If f =0, then the amplitude coefficients in A are specified 
to be constant. Update the template t with the solution of a subproblem: 



min— f-A„t„ +Za„ t„ subjectto u^ t M 

t 2 II Pi Pl \\2 P'll Kill j n v 



(19) 



5) Represent f asT a Update the amplitude coefficient vector a with the solution of 



a subproblem: 



1 II m2 || || 

min— f-T„ a„ +Aa„ a„ subjectto a„ >0 

a ? II 112 ^ z ' II ^ z ' 112 ^' 



(20) 



where /7 a > ,r denotes that the all elements of the vector a is positive. 
6) If i < M, update i <— i + 1 and go to step 3). If i = M and the stopping criterion is not 
satisfied, go to step 2). 



For stable computation, the update stage of the amplitude coefficient in Step 5) is omitted 
when the /2-norm of the template t becomes zero after Step 4). 

The closed form solution of (19) is 



ll 2 - Aa P, 



-v for \\v\\ 2 >Aa p . 



for ||v|| 2 < Aa 



where 



The solution of (10) is 



v = A r — 

Pi Pi 



<K r J 



11 2 "Pi 



Pi 112 



|w 


\ 2 ~^ a pi 




IHI2 







^ 





/or |w|| > Aa 



/or |w| 2 < 1^. 



where 



=fe,.) + 



(21) 



(22) 



(23) 



(24) 

(•)+ denotes replacing the negative elements of a vector with zero. The both solutions of the 
subproblems guarantee the decrement of the cost E. Thus, the cost E decreases until 
convergence. However, the set of the resultant periodic subsignals after the convergence of 
the iteration does not always obtain a minimum of the cost function E exactly. If any 
periodic subsignal becomes zero in iteration, the amplitude coefficients are specified to be 
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constant in step 4) of the next iteration. The proper search direction for t may not be 

obtained by these amplitude coefficients. However, the h norms of the periodic signals that 
eliminated by the shrinkage in (21) and (23) is small enough to approximate the signal. 
Hence, we accept the periodic subsignals obtained by this algorithm as the result of the 
sparse decomposition instead of the proper minimiser of the cost E. 



Tested set 



28, 44, 52 
30, 31, 32 
50, 51, 52 



Ave. Std. Dev 



14.6, 16.6, 12.6 2.4, 2.4, 2.2 
16.9, 21.0, 20.7 3.1, 2.7, 2.7 
10.8, 12.7, 10.8 1.7, 1.9, 1.7 



Table 1. SNR improvements (dB) obtained by the sparse periodic decomposition for 
mixtures of three periodic signals. 

5. Decomposition examples 

In this section, we provide several examples of the sparse periodic decomposition. The 
examples demonstrate the decomposition of synthetic signals generated by adding three 
periodic signals. The length of the mixture and three source signals N is 256. Each source 
signal is generated with the model for the periodic signals shown in (1). Each waveform 
within a period is generated by Gaussian random variables. The average of the waveform of 
a period is normalized to zero. The amplitude envelope of one of the three source signals are 
specified as a constant. The envelopes of the other two source signals are specified as a 
decreasing Gaussian function 

for n>0 



a(n) = exp 
and an increasing Gaussian function 

a(n) = exp 



2(n-N) 

N 



for n>0, 



respectively. The squared norms of the three source signals are normalized to unity. Since 
the three source periodic signals can be assumed to be independent to each other, the SNR 
of each source signal in the mixture is about -3.0 dB. The sets of three periods for mixtures 
are shown in the first column of Table. 1. The first set contains the periods have three 
divisors. The second and third consist of closely spaced periods. An example of the mixture 
is shown in Fig. 1(a). The three source periodic signals are shown in Fig. 1(b), (c) and (d), 
respectively. 

For the sparse periodic decomposition, the sequence of the parameters {a p } p( =p and the 
sparsity parameter A have to be specified. The shrinkage of the k-norm of the periodic 
component in the decomposition algorithm is performed with the threshold Xa v in (21) and 
(23). The periodic signal i v with the k-norm that is less than the threshold is eliminated by 
the shrinkage. Obviously, if the residual r in (18) can be assumed to be a noise that is small 
enough to approximate the input signal, its periodic approximation has to be eliminated 
during the decomposition. We assume that the noise as a Gaussian noise with a variance a 2 . 
The product Xa v is specified as proportional value to the expected h norm of the 
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approximated Gaussian noise with the periodic signal model. The expected h norm of the 
periodic signal i v that approximates a Gaussian noise, of which envelope is constant, is 
approximated as 



-0.2 
-0.3 



-0.2 
-0.3 
-0.4 : 



£ ii f 4h CT > +1+ 



N 



(25) 









(a) Mixture 




100 150 200 

(b) Source signal (p=28) 
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(c) Source signal (p=44) 
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50 100 150 200 



(d) Source signal (p=52) 

Fig. 1. (a) Example of mixture of three periodic signals, the source periodic signals, (a) p = 28, 
(c) p = 44 and (d) p =52. 



The product Xa v is hence specified to a value that is proportional to this expectation. In 
actual decomposition, a is assumed to be 1% of the k-norm of the input signal. Aa p is 
specified as the expectation shown in (25). 

In the experiments, we supposed that the period of the source signals are integer in the 
range [10, 59]. The periods for the decomposition are also defined as integers in this range. 
So, the number of the periodic signals that are obtained by the decomposition is 60. The 
iteration of the decomposition algorithm explained in Sect. 4. 2 is stopped when Zoo-norm of 
the difference of the periodic signals before and after updating is lower than a threshold 
value. The threshold is specified as 0.01 xAa p for all experiments. 

In order to evaluate the decomposition, we compute the improvement in SNR. The 
improvement in SNR is computed as the difference of the SNRs of the mixture and 
decomposition results for each source period. We generate 1,000 mixtures to test the 
decomposition algorithm for each set of periods. Table 1 shows the averages and standard 
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deviations of the SNR improvements of the decomposed periodic signals for 1,000 tests. The 
average SNR improvements of the decomposition results exceed 10 dB. By these results, we 
see that the proposed decomposition can obtain significant decomposition results and 
separate three sources into its periods. In Fig. 2 and 3, an example of the mixture and its 
decomposition result are shown. The discrete Fourier transform (DFT) spectrum of the 
mixture (Fig. 1(a)) is shown in Fig. 2(a). 
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(b) 12-norms of the decomposed periodic signals. 

Fig. 2. (a) DFT spectrum of the mixture in Fig. 1(a) and (b) distribution of the h norm of the 
decomposed periodic signals. 
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(a) Decomposed signal (p=28) 
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(b) Decomposed signal (p=44) 
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(c) Decomposed signal (p=52) 

Fig. 3. Decomposed periodic signals, (a) p =28, (b) p = 44 and (c) p =52. 



The distribution of h norm of the resultant periodic signals of the mixture is shown in Fig. 
2(b). As seen in Fig. 2(b), three periodic signals with large amplitude appear at the source 
periods. Small harmonics components are separated from the source periods due to the 
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weighting of the sparsity penalty, however, the almost energy of the mixture is decomposed 
into the three source periods. In Fig. 3, the periodic signals that appear in the decomposition 
result are also shown. In this set of the periods, the harmonics with periods 1, 2, and 4 which 
are the common divisors of the source periods cannot be separated accurately. However, the 
other harmonics are well collected to three fundamental periods. 
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Fig. 4. (a) Speech signal (male, duration: 8.1 s, sampling freq. 
energy distribution of (a). 
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Fig. 5. (a) Speech signal (female, duration: 8.1 s, sampling freq. : 8 kHz) and (b) time-period 
energy distribution of (a). 



6. Application to speech representation 

In the synthetic signal examples, the signal mixtures consist of source periodic signals with 
integer periods. However, periods of many periodic signals that include speech and acoustic 
signals are not integer. In order to examine the sparse periodic decomposition for the signals 
with non-integer periods, we apply the proposed sparse decomposition to speech mixtures. 
The speech signals for the experiments were selected 3 Japanese male and 3 female 
continuous speeches of about 8 s taken from ATR-SLDB (Spoken Language Database). The 
sampling rate of each speech signal is converted to 8 kHz. 15 speech mixtures that consist of 
two different speeches that are normalized to same power are generated. 
For periodic decomposition, each mixture is divided into segments that contain 360 samples 
with 3/4 overlap. In each segment, the periods for decomposition are specified to be 
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integers in the range [10, 120] which corresponds to the range of the fundamental 
frequencies of most men and women. The stopping rule of the iteration of the relaxation 
method and the parameters are specified as the same rule that is mentioned in Sect. 5. 
The examples of the male and female utterances and its time-period energy distributions are 
shown in Fig. 4 and Fig. 5, respectively. In Fig. 4(b) and 5(b), the brightness indicates the 
power of the resultant periodic signals for each segment and period. Darker pixels indicate 
higher powers of the resultant periodic subsignals. 



<» 20000 



< -10000 




Fig. 6. (a) Mixture of female and male speeches and (b) time-period energy distribution of (a) 



Speakers 



(M,M) 
(F,M) 



Ave. SNR Min. SNR 



Max. SNR 



Ave. num. of 
periods 



20.1 
20.2 
20.2 



10.4 
10.3 
10.2 



28.9 
27.6 
28.9 



16.1 
11.2 
14.0 



Table. 2. Average, minimum and maximum SNRs (dB) of approximated speech segments 
and average numbers of periodic signals obtained by the sparse decomposition 



Our method decomposes a signal into the periodic signals with only integer periods. Under 
this limitation, the speech components with non-integer periods and the frequency 
variations that occur in a segment are represented as the sum of some periodic signals. So, 
we see that the pitch contours are represented by some neighbouring periods in these time- 
period distribution. Moreover, small periodic components with periods that are multiples 
and divisors of the fundamental periods appear. These periodic components appear due to 
the non-integer periodic components of the speeches and the weighting of the sparsity 
measure in (17). However, the most of the signal energy is concentrated around the 
fundamental pitch periods of the speeches. 

We also show the time-period energy distributions of the mixture of two speeches. Fig. 6(a) 
and (b) show the mixture of the source speech signals shown in Fig. 3(a) and Fig. 4(a) and its 
time-period energy distributions, respectively. We see that the time-period energy 
distribution of the mixture in Fig. 6 is almost equal to the sum of the two distributions of the 
source speeches shown in Fig. 4(b) and Fig. 5(b). The both of the pitch contours of the two 
source speeches are preserved in the distribution of the mixture. The proposed 
decomposition method can approximate the mixture while concentrating the energy of each 
speech to its pitch periods and provides sparse representation of the mixture. It is expected 
that the pitch periods of both the speech signals will be tracked in this time-period energy 
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distribution. Moreover, speech separation will be achieved by assigning the resultant 
periodic signals to the sources. 

In order to evaluate the approximate decomposition, we compute the SNR and the number 
of the non-zero resultant periodic signals for each segment where the h norm is greater than 
the noise level. The average, maximum and minimum SNRs over all voice active segments 
of mixtures are shown in Table 2. In this table, F an M denote female and male source 
speeches, respectively. The average numbers of periods for approximation of a segment are 
also shown. We see that the average approximation precision of the proposed 
decomposition is about 20 dB in the segmental SNR. The average number of the periods 
yield by the decomposition is about 14 for segments of speech mixtures consist of two 
speeches. 



Speaker 


Proposed 


DFT 


Proposed 




(with sources) 


(with sources) 


(with ref. sig.) 


(M,M) 


9.9+0.6 


13.5+0.5 


3.9+1.0 


(F,F) 


9.5+0.3 


13.5+0.5 


3.2+0.9 


(F,M) 


F: 10.1+1.5 


F: 14.4+1.0 


F: 6.5+2.5 




M: 9.8+1.0 


M: 14.3+1.0 


M: 6.7+2.7 



Table 3. Average SNRs (dB) of separated speeches. 

Next, we demonstrate the speech separation from a mixture with the sparse periodic 
decomposition. In this experiment, the speech separation is performed by assignment of the 
resultant periodic subsignals to the sources in each segment. 

First, we use the clean source signals for assignment of the resultant periodic signals. 
The separation is carried out by the following steps for each segment: 

1. The segment of the mixture is decomposed into the set of the periodic signals {f p } pe p. 

2. The normalized correlations between the resultant periodic signals and the clean source 
segments {s;}/ = i, 2 are computed. 

3. Each resultant periodic signal f p are added to the separated output that is associated 
with the z-th source s? that obtains larger correlation. 

For recovering source signals, each resultant periodic signal is multiplied with a Hanning 
window in each segment. This assignment method does not obtain optimum separated 
results in terms of the SNR exactly. However, this experiment gives the rough ideal 
performance of the source separation by using the proposed sparse decomposition. 
For comparison, the ideal separation results that are obtained by a STFT that is widely 
utilized for the sparse representation of speech signals are demonstrated. In the separation 
with the STFT, the ideal binary masks [20] are computed from the clean source speeches. 
The mixture and the source signals are segmented by 512 points Hamming window with 
3/4 overlap. In each segment, the DFT spectrum of the mixture and the source signals are 
computed. Each frequency bin of the DFT is assigned to the source whose amplitude of the 
frequency bin is larger than the other. The separation results obtained by the proposed 
decomposition and the DFT are shown in Table 3. 

In this table, the SNRs of the separated speech signals are shown. We see that the SNRs of 
the separated speeches obtained by the proposed method are lower than the DFT by about 
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4dB. In the separation obtained by the proposed method, the approximation errors caused 
during the decomposition are involved in the separated output. Since the frequency 
resolution of the periodic decomposition is lower than the DFT at high frequency bands, the 
interferences between two speeches mainly occur at high-frequencies. However, the 
proposed representation is sparser than the DFT spectrum. In this experiment, the DFT 
yields 257 frequency bins for each segment. So, the DFT based separation is the problem of 
the assignment of the 257 frequency bins. In contrast, the average number of the periodic 
signals yield by the proposed method is about 14 for a segment. Comparing the proposed 
decomposition with the DFT, the separation problem can be reduced to relatively small size 
of a combinatorial optimization by the proposed decomposition. 
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(b) 

Fig. 7. Separated speech signals obtained by sparse periodic decomposition with reference 
speeches, (a) separated male speech (SNR: 7.2dB) and (b) female speech (SNR: 7.1dB) from 
the mixture shown in Fig. 5(a) 

In above separation experiments, we assume that the source speeches are known. Next, we 
demonstrate the single-channel speech separation by referencing the clean speech segments. 
In this scenario of the separation, two speakers in a mixture are known and the clean 
speeches of the speakers are available, but the contents of the speeches in the mixture are 
unknown. In order to assign the periodic signals to the sources, a set of the clean speech 
segments of the z-th speaker is defined as {ci f j]i<j<Nr where Nr is the number of the reference 
segments. 

The resultant periodic signal i v is assigned to the z-th speaker that gives the maximum of the 
normalized correlation as: 



max t 



hi f kv ■ 

For this experiment, segments that are generated from a clean speech of 20 s are used for the 
references of each speaker. The segments where the voice is not active are rejected from the 
references. The references do not include the source utterances in the mixtures. The SNRs 
obtained by the separation with the references are also shown in Table. 3. Obviously, such a 
simple separation method causes many false assignments. For separation of the mixture 
consists of the speakers of same gender, the averages of the improvements of SNR are lower 
than 4dB. However, the averages of SNR close to the ideal results and are about 6.5dB for 
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the speakers of opposite gender. The separated signals from the mixture in Fig. 6(a) are 
shown in Fig. 7(a) and (b). 

The single channel speech separation methods based on frequency masking of spectrum 
have been proposed [12] [13] [14]. In these methods, statistical models for the frequency 
spectra of the speakers are preliminary learnt. The separation is performed on the frequency 
spectrum of the mixture by using the statistical models. In our approach, the proposed 
sparse decomposition yields the small number of the periodic signals which approximate 
the source signal due to the sparsity penalty. So, the separation of two speeches that have 
less similarity can be performed by such a lazy assignment method. 

7. Conclusions 

In this chapter, we present a sparse decomposition method for periodic signal mixtures. The 
proposed decomposition is based on the model for the periodic signals with time-varying 
amplitude and the sparsity of the periods that appear in the decomposition result. In 
decomposition experiments of the synthetic signal and the speech mixtures, we 
demonstrated that the proposed decomposition has the ability of source separation. 
The assignment method that is employed for the single-channel speech separation 
demonstrated in this paper is too simple to obtain good separation results. In our 
decomposition results, as seen in the figures in Sect. 4, the speech pitch contours are 
involved. We can use the temporal continuity of the speech pitches and spectra over the 
consecutive segments for improvement of the accuracy of the assignment. The accurate and 
robust assignment of the decomposed periodic signals is a topic for future research. 
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1. Introduction: magnetic resonance spectroscopic (MRS) signals 

A magnetic resonance spectroscopic (MRS) signal is made of several frequencies typical of the 
active nuclei and their chemical environments. The amplitude of these contributions in the 
time domain depends on the amount of those nuclei, which is then related to the concentration 
of the substance (Hornak, 1997). 

This property is exploited in many applications of MRS, in particular in the clinical one. The 
MRS spectra contain a wealth of biochemical information characterizing the molecular content 
of living tissues (Govindaraju et al., 2000). Therefore, MRS is a unique non-invasive tool for 
monitoring human brain tumours, etc. (Devos et al., 2004), if it is well quantified. 
When an MRS proton signal is acquired at short echo-time (TE), the distortion of spectral mul- 
tiplets due to J-evolution can be minimized and the signals are minimally affected by trans- 
verse relaxation. Such signals exhibit many more metabolite contributions, such as glutamate 
and myo-inositol, compared to long TE spectra. Therefore, an MRS signal acquired at short 
TE presents rich in vivo metabolic information through complicated, overlapping spectral sig- 
natures. However, it is usually contaminated by water residue and a baseline which mainly 
originates from large molecules, known as macromolecules. As the shape and intensity of the 
baseline are not known a priori, this contribution becomes one of the major obstructions to 
accurately quantify the overlapping signals from the metabolites, especially by peak integra- 
tion, which is commonly used in frequency-based quantification techniques. Also, by seeing 
only the frequency aspect, one loses all information about time localization. 
A number of quantification techniques have been proposed, which work either in the time 
domain (see Vanhamme et al. (2001) for a review) or in the frequency domain (see Mierisova 
& Ala-Korpela (2001) for a review). The time-domain based methods are divided into two 
main classes: on one side, non-interactive methods such as SVD-based methods (Pijnappel 
et al., 1992) and, on the other side, methods based on iterative model function fitting using 
strong prior knowledge such as QUEST (Ratiney et al., 2004; 2005), LCModel (Provencher, 
1993), AQSES (Poullet et al., 2007), or AMARES (Vanhamme et al, 1997). 
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However, there also exist techniques that analyse a signal in the two domains simultaneously 
and are therefore more efficient than, say, the Fourier transform, which gives only spectral in- 
formation. The result is a time-scale and or a time-frequency representation, such as provided 
by the wavelet transform (WT) and the Short-Time Fourier transform (STFT). In addition, both 
transforms are local, in the sense that a small perturbation of a signal which may occur during 
the data acquisition will result only in a small, local modification of the transform. 
A number of wavelet-based techniques have been proposed for spectral line estimation in 
MRS, including the continuous wavelet transform (Delprat et al., 1992; Guillemain et al., 1992; 
Serrai et al., 1997) and the wavelet packet decomposition (Mainardi et al., 2002). Among the 
various possibilities, we will concentrate our discussion on the continuous wavelet transform 
(CWT) with the Morlet wavelet (MWT). All wavelet calculations have been performed by our 
own wavelet toolbox, called YAWTb (Jacques et al., 2007). Some of the experimental aspects 
have been reported in Suvichakorn et al. (2009). For the convenience of the reader we have 
collected in the Appendix the basic features and properties of the CWT. 

In the following sections, we will study the performance of the Morlet WT to retrieve parame- 
ters of interest such as resonances frequencies, amplitude and damping factors, for nuisances 
or impairments generally encountered in in vivo MRS signals: noise, baseline, solvent, and 
non-Lorentzian lineshapes. 

2. The Morlet wavelet transform 

The wavelet transform (WT) of a signal s(t) with respect to a basic wavelet g(t) is 



/aJ g \ a 



S(r,a) = — g[—-)s(t)dt 



= ^Va [G(aw)S(cv)e ic0T dw, (1) 

2tt J 

where S(cv) is the Fourier transform of the signal, a > is a dilation parameter that charac- 
terizes the frequency of the signal (since 1/a is essentially a frequency), t £ IR is a translation 
parameter that indicates the localization in time and G(acv) is the complex conjugate of the 
(scaled) Fourier transform of g(t). We can think of the basic wavelet as a window which slides 
through the signal, giving the information at instantaneous time r. The window is also dilated 
by a, so that a small a corresponds to a high frequency of the signal, and vice versa. As a re- 
sult, the WT becomes a function of both time and frequency (scale). For more details, see the 
Appendix. 

A technique based on the continuous wavelet transform (CWT) was proposed by Guillemain 
et al. (1992). By exploiting the ability of the CWT to see the information in the two domains 
simultaneously, it can extract the information from MRS signals directly without any decom- 
position or pre-processing, in order to quantify an MRS signal. The technique proceeds in two 
steps: (i) detection of the frequency of the peaks in MRS signals and (ii) characterization at 
each detected frequency. It can be described as follows. 

At a particular value of a, the WT S a (r) = S(x, a) can be represented in terms of its modulus 
| S a (t) I and phase <J> fl (r), namely, 

S e (T) = \S a {r)\^\ (2) 
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with an instantaneous frequency 
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)T Im[lnS a (r)] 
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(3) 



_S a (T) ^T 

Next, let us consider an MRS signal with a Lorentzian damping function, namely, 

s L (t) = Ae-^e^+rt <* S L (cv) = 2nAe i( PS(cv - (cv s + iD)) f (4) 

where D and q> denote the damping factor and the phase of the signal. Its WT is accordingly 



S L (r,a) = JaAe l( ?e- DT e lw s T G(a(w s + iD)) 

= ^fas(T)G(a(co s + iD)). (5) 

For a Morlet function scaled by a dilation parameter a (we omit the negligible correction term, 
see Eq.(A.9)), namely, 

G M (aco) = exp ( - \o- 2 (aco - o^o) 2 ), (6) 

it can be seen that the modulus of S(r,a) is maximum, i.e., ^-S(r,a) — >■ 0, when j^G —> 0. 
Given that a > and the assumption that cv s ^> D, the maximum can be found along the scale 
a r = coq/co s (this is called a horizontal ridge), which then gives 

aa r D x 



and consequently 



G M (a r (co s + iD)) = expl -j=- I , (7) 

S, r (T) = v^exp(^) s(r), (8) 

which is identical to the signal s(t) multiplied by a coefficient depending on the still unknown 
D. Consider the modulus of the Morlet wavelet transform (MWT) along a r , 

s(t)| 



|S«,(T)| = 


(cra r D\ 2 


ln|S flr (r)| = 


1, fcra r D\ 2 
2 lna+ {v2) 4 


D 


-^ln|S ar (r)|. 



+ lnA- Dr. (9) 

That is, 

(10) 
Knowing D can now lead to the estimation of the amplitude resonance A of the signal by 

A=\s(t)\e Dt . (11) 
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time (s) dilation parameter (a) 

(a) (b) 

Fig. 1. (a) Phase of the Morlet wavelet transform of a signal s(t) containing two frequencies 
cv s =32 and 64 rad/s and (b) its instantaneous frequency. Here a = 1,coq = 5 rad/s, sampling 
frequency F s = 256 s _1 , data length / = 1024 points. 



Since S Ur (t) is a function of time, the derived D is also a function of time. This is beneficial for 
analysing signals that do not have a steady damping function. In addition, considering the 
phase of the MWT along a r , namely, 



argS flr (T) 



we also have 






dr 



: o; s T + < 



argSa r (T) 



O ar (r), 



(12) 



as in Eq.(3). Strictly speaking, the instantaneous frequency at the scale a r of the Morlet trans- 
form is co s . This can be observed in Figure 1, which shows that the instantaneous frequency 
intersects the line coq/u at a = coq/cv S/ where co s =32 and 64 rad/s are the frequencies of the 
signal. The phase of the signal cp G ( — n, n) can also be derived from the phase of the WT, if 
needed. The property given in Eq.(12) is useful for analysing an n-frequency signal; it indi- 
cates the actual frequencies of the signal and the scale a that we should consider. In addition, 
if its frequencies are sufficiently far away from each other, so that G(aco) treats each spectral 
line independently (Barache et al., 1997), the amplitude at each frequency can thus be derived. 
When two frequencies are very close to each other (this also depends on the sampling fre- 
quency), increasing the frequency of the Morlet function coq can better localize and distinguish 
the overlapping frequencies. On the other hand, co s can be obtained iteratively by 

1. Initializing a = U{ at some values. 

2. Calculating the instantaneous frequency, namely Cl ai . 

3. Assigning the new value to a z+ i = coo /Cl ar 

4. Repeating the process until a converges to co s . 

Figure 2 illustrates an overlap of two frequencies and the derived instantaneous frequencies 
using the iteration method. The derived frequencies converge to the true frequencies within a 
few steps. 
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dilation parameter (a) 



0.11 0.12 



(c) 

Fig. 2. (a) The MWT of y(t) = exp(i55t) + exp(z'60f) and (b) its instantaneous frequencies 
when using the iterative method. Here a = 1,coq = 5.5 rad/s, F s = 800 s _1 , / = 1024 points, 
(c) Comparison of the instantaneous frequencies by the non-iterative and the iterative method. 
The symbol o indicates an initial value of a. 



3. Continuous Wavelet Transform and the in vivo MRS challenges 

3.1 Gaussian White Noise 

An in vivo MRS signal is always impaired by additive noise, which is usually assumed to be 
white gaussian. This noise causes oscillations in the instantaneous frequency derived with 
the CWT representation, as illustrated in Figure 3 which shows the instantaneous frequency 
derived from a signal with a peak at a frequency of 32 rad/s with an additive Gaussian noise 
corresponding to a signal to noise ratio (SNR) of 10. 1 In order to reduce this effect, Guillemain 



1 The Signal to Noise ratio SNR is defined as the ratio of the time domain first point amplitude of the 
resonance to the time domain noise standard deviation 
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Frequency (rad/s) 

(a) 



0.2 0.25 

dilation parameter (a) 



(b) 



Fig. 3. (a) A spectrum with one resonance at 32 rad/s with SNR=10 (cr n = 0.079) and (b) 
its instantaneous frequency derived by the Morlet wavelet at t = 4.7 s (coq = 5 rad/s, a=l, 
Fs = 800 s" 1 ). 




translation (seconds) 



averaging time (s) 



(a) (b) 

Fig. 4. For the signal shown in Figure 3(a): Derived Lorentzian damping factor and (b) ab- 
solute frequency estimation error with respect to the averaging time, calculated at the scale 
a = cvo/cos of the Morlet wavelet transform (SNR = 10, cv s = 32 rad/s,o;o = 5 rad/s, cr = 1, F s = 
800 s" 1 ). 



et al. (1992) suggested averaging in time the derived parameters, for instance O fl (r), i.e., 

T 



a 



1 -JTq 



d a (r)dr. 



(13) 



As can be seen in Figure 3, averaging in time reduces the noise effect on the derivation of 
the instantaneous frequency. 2 One can see that averaging creates many steady points. At the 
scale a = cvq/32, the instantaneous frequency is about, but not exactly, 32 rad/s. Here, the 



■ This property might be used for denoising, but this has not been exploited. 
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Fig. 5. (a) The Fourier transform of a 1056-rad/s signal with baseline; b) Its instantaneous 


frequ 
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10 rad/s, a = 1). The baseline is modelled by a cubic spline. 







averaging time is 1.56 s. Figure 4(b) shows the evolution of the absolute frequency estimation 
with respect to the averaging time. Increasing the averaging time is likely to decrease the 
estimation error, as illustrated in Figure 4(b). The same approach can be used to derive the 
instantaneous damping factor. The estimated instantaneous damping factor is also smoother 
and closer to the actual damping factor when time averaging is employed. Although the 
method described above should work at any value of a, there is a particular range of a that is 
meaningful, and should be wisely selected. As a rule of thumb, this range should not be far 
from the scale that maximizes the modulus of the Morlet WT. 



3.2 Baseline 

The baseline corresponds to contributions from large molecules, with a broad frequency pat- 
tern in the MRS spectrum. Thus, it becomes a major obstruction in the quantification of 
metabolite contribution from the MRS signals. First, we simulate the baseline by cubic splines 
in order to study the performance of the MWT when a baseline is present. In the case of 
Figure 5, the simulated baseline has no effect on the instantaneous frequency derived from 
the WT. Then, we used a baseline modelled with 50 randomly distributed Lorentzian pro- 
files with a large damping factor, compared to the signal-of-interest at 3447 rad/s, e.g. Si (t ) = 
exp(-lQi) exp(z'3447£) + B(t) where B(t) = exp(-50f)[0.2exp(z'3447f) +0.3exp(i2000f) + . . .] 
is the baseline (see Figure 6). The first component of B(t) has the same frequency as the sig- 
nal, in order to imitate the overlap between the baseline and the signal. It is found that the 
modelled baseline does not prevent an accurate estimation of both the damping factor and 
the amplitude derived from the Morlet WT, provided one waits until both the effect of the 
baseline and the edge effect (discussed in Section 4.1 below) have died out. In the example 
shown here, the waiting time is approximately 0.2 s. 

The MWT in Figure 6(b) tells us that the baseline affects only the beginning of the transform in 
the time (t) axis, comparing to the long, clear peak of our 3447-rad/s signal. This means that 
the baseline can be assumed to decay faster than the pure signal, and the method described 
should still be effective without removing the baseline beforehand. Such an assumption has 
been widely used in spectroscopic signal processing, where several authors have proposed 
truncation of the initial data points in the time domain, which are believed to contain a major 
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Fig. 6. (a) The Fourier transform of a 3447-rad/s Lorentzian signal with baseline. The latter is 
modelled by large Lorentzian damping factors; (b) Its Morlet WT and the derived parameters: 
(c) damping factor and (d) amplitude. The actual parameters are 10 s _1 and 1 a.u. for the 
damping factor and amplitude, respectively, (coq = 100 rad/s, a = 1). From Suvichakorn 
et al. (2009). 



part of the baseline. However, some information of the metabolites could be lost and a strat- 
egy for properly selecting the number of data points is needed (see Rabeson et al. (2006) for 
examples and further references). 

Next, in order to study the characteristics of the real baseline by the Morlet wavelet, an in 
vivo macromolecule MRS signal was acquired on a horizontal 4.7T Biospec system (BRUKER 
BioSpin MRI, Germany). The data acquisition was done using the differences in spin-lattice 
relaxation times (Tl) between low molecular weight metabolites and macromolecules (Behar 
et al, 1994; Cudalbu et al, 2009; 2007). 
As seen in Figure 7, the metabolite-nullified signal from a volume-of-interest (VOI) central- 
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Fig. 7. (a) The signal of baseline + residual water (a) in time domain; and (b) in frequency 
domain. 
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Fig. 8. (a) Frequency response of creatine at 4.7 Tesla and (b) its Morlet WT {coq = 10 rad/s, 
a = 1, F s = 4006.41 s _1 ). The parameters derived from the Morlet transform are D = 10 s -1 , 
co 1 = 1056 rad/s, A 1 = 1330 a.u. and cv 2 = 2168 rad/s, A 1 = 1965 a.u. 



ized in the hippocampus of a healthy mouse 3 resulted from a combination of residual water, 
baseline and noise. Compared to the simulated signal of creatine, whose frequency response 
and Morlet WT are shown in Figure 8, the signal decays much faster, making it suitable to use 
the Morlet wavelet to analyse the MRS signal as described earlier. For studying this, the two 
signals are normalised to the same amplitude and added together. Then the amplitude of the 



3 An Inversion-Recovery module was included prior to the PRESS sequence (echo-time = 20ms, repe- 
tition time = 3.5s, bandwidth of 4kHz, 4096 data-points) in order to measure the metabolite-nullified 
signal. The water signal was suppressed by variable power RF pulses with optimized relaxation delays 
(VAPOR). All first- and second-order shimming terms were adjusted using the Fast, Automatic Shim- 
ming technique by Mapping Along Projections (FASTMAP) for each VOI (3x3x3 mm 3 ). Inversion 
time = 700 ms. 
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creatine is derived with the Morlet WT. Next, we multiply the simulated, normalised creatine 

by 0.5, 1, 1.5, For each of these values, we derive the amplitude and plot the result in Figure 

9. The recovery of the (simulated) creatine at different amplitudes, after adding it to the base- 
line signal, reveals that the amplitude of the metabolite can be correctly derived using t = 0.4 
s, whereas at earlier time (t < 0.2 s) the derived amplitude still suffers from the boundary 
effect (we will discuss this effect in Section 4.1). However, the metabolite signal is covered 
later by noise (t = 0.77 s), giving an inaccurate amplitude estimate. Therefore, the time to 
monitor the amplitude of the metabolite should be properly selected. Another data set of the 
baseline 4 acquired at 9.4T, with a better signal to noise ratio and a better water suppression, 
shows similar characteristics (see Figure 10). 
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Fig. 9. Derived amplitude at cv = 1056 rad/s, using coq = 100 rad/s and a = 1 from a signal 
containing a simulated creatine signal and an in vivo acquired macromolecule signal. 

3.3 Solvent 

In MRS quantification, a large resonance from the solvent needs to be suppressed to unveil the 
metabolites without altering their magnitudes. The intensity of the solvent is usually several 
orders of magnitude larger than those of the metabolites. 



4 received from Cristina Cubaldu, Laboratory for Functional and Metabolic Imaging (LIFMET), Ecole 
Poly technique Federale de Lausanne (EPFL). 
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Fig. 10. 
Fourier 



4.7 Teslas 9.4 Teslas 

Macromolecules MRS signals acquired at 4.7 Teslas and 9.4 Teslas, respectively, their 
transforms and their Morlet WT. 



The Morlet WT sees the signal at each frequency individually, therefore it can work well even 
if the amplitudes at various frequencies are hugely different, which normally occurs when 
there is a solvent peak in the signal. In order to illustrate this, the Morlet WT has been applied 
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to the following signal 
s(t) = lOOe 



-S5t e i32t . e -1.5t e i60t , g-OSt^Ot , g -f g *120* , g -2* g *150* 



(14) 



as seen in Figure 11 (a). This signal has an amplitude of 100 at 32 rad/s and 1 elsewhere. The 
high amplitude can affect other frequencies if they are close to each other. This is illustrated in 
Figure 11 (b) when a Hann window is applied to the signal in order to separate each frequency. 
Using the aforementioned method, the amplitude of 1 is derived as 0.980, 0.911, 0.988 and 
0.974 respectively. The error ranges within 1.2-8.9 %, without any preprocessing. 
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-extracted sprectrum 





50 100 150 200 250 300 350 400 
frequency (rad/s) 



(a) (b) 

Fig. 11. (a) The Fourier transform of a signal with different amplitudes and the spectrum 
extracted by the Morlet wavelet and (b) by a Hann window. 



3.4 Non-Lorentzian lineshape 

The ideal Lorentzian lineshape assumes that the homogeneous broadening is equally con- 
tributed from each individual molecule. However, imperfect shimming and susceptibility 
effects from internal heterogeneity within tissues lead to non-Lorentzian lineshapes in real ex- 
periments (Cudalbu et al., 2008). These effects are typically modelled by a Gaussian lineshape 
(Franzen, 2002; Hornak, 1997). Since the inhomogeneous broadening is often significantly 
larger than the lifetime broadening, the Gaussian lineshape is often dominant. If the line- 
shape is intermediate between a Gaussian and a Lorentzian form, the spectrum can be fitted 
to a convolution of the two functions (Marshall et al., 2000; Ratiney et al., 2008). Such lineshape 
is known as a Voigt profile. 

Next we will explore how the Morlet WT can deal with the Gaussian and Voigt lineshapes. 
Consider a pure Gaussian function modulated at the frequency co S/ namely, 



s G (t) = Ae-r* 



(15) 



Its Morlet WT is 



S G (r,a) 



/a J 



y/a 



gM [ —^~ )s G (t)dt 



-7t 2 e icv s t e -(^- a ) e -iw Q {^-) dt 



A 



f e -(ht 1 +k 2 t+h) dt/ 



(16) 
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where 

1 
fcl =7 + 



2a 2 a 2 



k 2 = -i(cv, 

.COnT t 2 

k 3 = -i-^- + 



a ' 2(T 2 a 2 ' 
Eq.(16) is known as a Gaussian integral and can be computed explicitly: 

r e -(hf+k 2 t+h) dt = /z e |-fe. (17) 

J-oo V M 

As a result, the Morlet WT at the scale a r = coo/co s is 

S G , ar (r)=k 4 Ae- k ^ 2 e i ^, (18) 

where 

h 
h 



2n(2ya 2 a 2 + 1) 
7 



27(7 2 fl 2 + 1 ' 

which is also a Gaussian function at the frequency co s . The width and amplitude of this new 
Gaussian function are functions of co s and of the width of the original Gaussian signal Sc(t). 
Therefore, similarly to the process of the Lorentzian lineshape, the amplitude {A) and the 
width of the Gaussian function (inversely proportional to 7) can be obtained as follows: 

1. Findo; s = ^ argS G/ar (r). 

2. Find 7 from the second derivative of In \SQ /Ur (r) |, which yields 

y = -, ^w • (19) 

^ln|S GA (T)|) + <*£ 

3. Find A from the calculated to s and 7. 

On the other hand, the Morlet WT at the scale a r = cvq/cv s of a Voigt lineshape, 

s v (t) = Ae-^ 2 e- Dt e icVst f (20) 

is given by 

Sv,a r (t) = k 6 Ae~ k ^ T - k rf e ico * T , (21) 

where 

-D 2 

k 6 = k^e ^ 

•»-£• 

27 
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Fig. 12. (a) The modulus of the Morlet WT (coq = 15 rad/s) of a signal of a frequency 60 rad/s 



with (a) undamped s(t 
and (d) Voigt s(f) 



e lb{jt ; (b) Lorentzian s(£) 



, (c) Gaussian s(t) 



e -t e -f e i60t 



lineshape. 



That is, at the scale a r , the Morlet WT of the Voigt lineshape is also a Gaussian function with 

the same width, but shifted in time, with the amplitude smaller than that of the Gaussian 

lineshape, and its instantaneous frequency is also equal to co s . 

Note that the scale a r = u?q/co s does not give exactly the maximum modulus of the WT. 

However, as seen in Figure 12, the modulus of the Morlet WT of a signal with a Lorentzian 

lineshape or a Gaussian lineshape (and also a Voigt lineshape) are maximal at the same scale 

a r , provided that a G IR and cv s ^> D. 

Figure 13 shows that the second derivative of the modulus of the Morlet WT can be used to 

describe the second-order broadening of the lineshape, no matter whether it is Gaussian or 

Voigt. In the case of a Voigt lineshape, 7 actually gives back a Lorentzian whose damping 

factor is obtained by Eq.(10). 
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Fig. 13. The Gaussian damping factor derived from the pure Gaussian signal and the Voigt 
signal considered in Figure 12 
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Fig. 14. (a) The comparison of the derived instantaneous frequency of the Morlet WT of 
a signal of a frequency 60 rad/s with different lineshapes, e.g. Lorentzian s(t) 
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4.7 s. Panel (b) shows the modulus of the Morlet WT of each line at a r 
coq =15 rad/s, F s = 800 s _1 , 1 = 1024 points. 



Kubo's lineshape 

The interaction between the Lorentzian and Gaussian broadening of lineshape depends on 
the time scale. For example, if the relaxation time (T2) is much longer than any effect modu- 
lating the energy of a molecule, the lineshape will approach the Lorentzian lineshape. On the 
contrary, if T2 is short, the lineshape is likely to be Gaussian. In order to account for this time 
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Fig. 15. J^ In | Sc,a r (t) I with respect to Kubo's 7 for the pure gaussian signal given in Eq.(15), at the scale 
a r — too/ to s . We have put oc — y/g, where 7 and g are the two parameters of the Kubo lineshape defined 
in Eq.(22). 



scale, Kubo (1969) uses a so-called Gaussian-Markovian modulation, namely 



8(0 



Lexp I 



-7*. 



r 



1 + 7O: 



(22) 



The parameter 7 is inversely proportional to T2 and g is the amplitude of the solvent-induced 
fluctuations in the frequency, li oc = y I g <C 1, the lineshape becomes Gaussian, whereas 
oc ^> 1 leads to Lorentzian. The width of the lineshape is £ 2 7. 

Solving Eq.(22) seems to be complicated, though may be possible. However, it turns out that 
the maximum modulus of the Morlet WT of a Kubo lineshape at co s = 60 rad/s occurs also at 
the scale a r = cvq/cv S/ like those of the Gaussian and Lorentzian lineshapes. In addition, the 
instantaneous frequency is still able to derive the to s , even better than the Gaussian lineshape, 
as shown in Figure 14(a), although the amplitude is broader than those of the Lorentzian, 
Gaussian or Voigt profiles, as shown in Figure 14(b). The damping parameters can also be 
derived by the linear relation between j^ In \SQ iUr (t) | and 7, as seen in Figure 15, whereas oc 
is related directly to J^- In | SQ Ar (r) | . 

4. Limitations of the Morlet wavelet transform 

In the previous section, the Morlet WT shows its potential for analysing an MRS signal by 
means of its amplitude and phase, in addition to its time-frequency representation. However, 
these techniques can be applied to well-defined lineshapes only. Another limitation is the 
requirement of a proper coq that should distinguish the signal from the solvent, but should not 
introduce noise in the result. In this section, we will look further on some more limitations 
that prevent the use of the Morlet WT to quantify MRS signals directly. 
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4.1 Edge effects 

Errors in the wavelet analysis can occur at both ends of the spectrum due to the limited time 
series. The region of the wavelet spectrum in which effects become important 5 increases lin- 
early with the scale a, thus it has a conic shape at both ends, as already seen in Figure 1(a) 
(see also the Appendix). The size of the forbidden region, which is affected by the boundary 
effect, varies with the frequency cvq of the Morlet wavelet function and the ratio between the 
frequency of the signal (co s ) and the sampling frequency (F s ). Figure 16 shows that the size 
becomes larger for a large cvq and low co s /F s . In practice, the working region is chosen so that 
the edge effects are negligible outside and the characterization of the MRS signals should be 
made inside this region, disregarding the presence of the macromolecular contamination. 
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3 150 
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co/F 



Fig. 16. Lines showing the width (in number of sample points) of the forbidden regions where 
the boundary effect becomes important, as a function of coq (rad/s) and the ratio between the 
signal frequency (o; s ) and the sampling frequency (F s ). From (Suvichakorn et al., 2009). 



4.2 Interacting/overlapping frequencies 

If two frequencies of the signal are close to each other, the wavelet can interact with both 
of them at the same time. This was already observed in Figure 2(a). Barache et al. (1997) 
suggested the use of a linear equation system to solve the problem. In the sequel, the simu- 
lated N-Acetyl Aspartate (NAA) is used to illustrate how the problem could be solved. The 
spectrum of the NAA, shown in Figure 17(a), is composed of two different regions, the high, 
single peak (NAA-acetyl part) and a group of overlapping frequencies (NAA-aspartate part). 
By using a high coq to separate the overlapping frequencies, the Morlet WT reveals that there 
are eight frequency peaks in the group as seen in Figure 17(b). The damping factors of the 
two parts of NAA are shown in Figure 18(a). Applying Eq.(10) directly to each peak causes 
an oscillation in the derived damping factor, compared to the smooth and stationary damping 



5 defined as the e-folding time for the autocorrelation of wavelet power at each scale. 
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Fig. 17. NAA : (a) Frequency response; (b) Its Morlet wavelet transform for coq 
(left) and cv = 500 rad/s (right). From (Suvichakorn et al., 2009). 



100 rad/s 



factor of the single peak. The size and frequency of the oscillation depends on the numbers 
of neighbours of each peak and the spectral distance to these neighbours. A proper damping 
factor can be achieved by averaging these oscillations in time. 
Next, we will try to derive the amplitude of each peak. Let us consider an MRS signal com- 



posed of n Lorentzian lines s(t) 



o-Dt 



LnSn(t), where s n (t) = A n e iw ^^ and n = 1,2, 



is an indexing number. Its Morlet WT gives local maxima close to the scales a\ = coq/coi, 
a 2 = coq/coi, and so on. Therefore, we can establish a systematic relation between S Ur and 
s n (t) at each scale as follows: 



Mi) 

V^2 

Sa 3 (T) 
V«3 



exp(-DT)C 



si(r) 
si(r) 

s 3 (t) 



where C = [C mn ] is a matrix with 



C nn = exp 

^■mn — exp 



a 2 alD 2 



2 2 






■CV n 



iD 



m 7^ n. 



The value of | C mn \ decreases when the resonating peaks are well resolved (no overlapping 
frequencies), in fact, it goes to zero when \w m — co n \ increases, independently of D. Also, 
| C mn | decreases when co m is high. If C mn is not negligible (overlapping frequencies), solving 
the linear equations gives the information for each s n (t). 
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Fig. 18. NAA: (a) Damping function derived by Eq.(10); (b) Amplitudes of NAA-aspartate 
part, derived by the linear equations (with zero phase). From (Suvichakorn et al., 2009). 



The damping parameter D for the equations can be derived by Eq.(10), although the over- 
lapping frequencies may cause oscillations in the solution, but these can be smoothened by 
averaging in time. 

There can be a bias from the estimation, depending on the number and distribution of overlap- 
ping frequencies, e.g. the distance between neighbouring frequencies and cvq. For the NAA 
(cv = 3447 rad/s), the bias is approximately 1% of its amplitude (in time domain), when coq 
= 200 rad/s is used. Note that Lorentzian lineshapes are assumed in these linear equations, 
and the result is presented in Figure 18(b). In case of non-Lorentzian lineshapes, the arbitrary 
damping function should be determined, and taken into account to solve the equation. 
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Fig. 19. In vitro measured Creatine at 9.4 T 
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4.3 Arbitrary lineshape 

Let us consider a signal with an arbitrary damping function D(t), namely, 

s(t) = AD(t)e( ic0st+( P\ (23) 

Its Morlet WT is defined by 

S(r,a) = -A^= f ' Dity^e-M^y \-™^)&t 
Ina^/a J 

= AQ J D(x + T)e iw * x e-M-a) e - iWQ (-«)dx; x = t-r, 

/ gj —i(iOQ—LOs)x 
J*[D(x + t)]J> 2a2 a 2 e a }dco (Parseval's theorem) 

-5=S(t,<i) = AC 2 [ F[D(x)]j™e^ aw +°''- *fd<v, 
y/a J 

where C\ = elcVsTf _ an d c 2 = (\/2n)~ 1 e l ^ CVsX ^ ( f ) \ When implemented (thus discretized), the 
equation above can be seen as the product of two matrices, namely, 

S = C 2 DG, 

and the damping function could be solved from the following equations 

AT[D(x)] = q^SG -1 , 

AD(x) =C 2 " 1 J 7-1 [SG _1 ], 
AD{t) = C^ l F- l [SG- l e iTCV ], 

where S is the matrix of the scaled wavelet coefficients, G is derived from the Morlet WT and 
the frequency-of-interest co s , and A is the unknown amplitude of the signal. For a combination 
of frequencies with the same damping function, dividing by \D(t) | should give us a possibility 
for comparing the amplitude at each peak relatively. 

5. Working in a real life environment 

By real life environment, we mean genuine acquired data, either in vitro or in vivo, rather than 
simulated ones. In that case, the ideal Lorentzian lineshape of individual peaks gets distorted. 
To give an example, we show in Figure 19 the analysis of an in vitro creatine signal. We see 
that intermittent noise appears, in the form of many disrupted, horizontal bands in the WT. 
Thus the noise occurs for a while at some particular frequencies and then disappears. 6 Such 
characteristics differ from the Gaussian white noise that usually appears as vertical bands in 
the WT. It is also possible that the Gaussian white noise at that duration has the same intensity, 
however. The analysis of this in vitro creatine signal shows that the frequency distribution 
at each peak is broad and the almost stationary Gaussian damping factor indicates that the 
acquired signal has a lineshape close to that of the Gaussian function. Nevertheless, deriving 
the amplitude using the Gaussian assumption may lead to an inaccurate estimation. 



6 We don't know the origin of that noise, which in fact represents the part of the signal that we cannot 
identify in terms of specific, known contributions. 
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When the acquisition is made in an in vivo environment, the exponential decay of an MRS 
signal is severely distorted. This is due to the inhomogeneity of the static magnetic field and 
to eddy currents induced in the magnet walls by switching magnetic gradient fields on and 
off. Apart from the problem of overlapping frequencies in each metabolite, an in vivo MRS 
signal is composed of several metabolite signatures. Therefore, the challenge is to find a good 
combination of the amplitudes that the Morlet WT derives at each frequency. Determining 
complete spectra of each metabolite is preferred to individual resonance. This is yet to be 
solved. 

Appendix:The mathematics of the CWT 

A.1. General definitions and properties 

The continuous WT is a mathematical tool which permits to decompose a signal in terms of 
elementary contributions called wavelets. A large body of literature exists for wavelet anal- 
ysis. We might refer the interested reader to the textbooks of Daubechies (1992), Torresani 
(1995), Ali et al. (2000), Antoine et al. (2004), or the elementary introductions (Antoine, 1994) 
and (Antoine, 2000). These wavelets are obtained from a single function g by translations and 
dilations, 

*™® = Ta*(cr)' (AJ) 

where the parameters of translation, r £ R, and dilation, a > 0, may be continuous or discrete. 
The CWT of a signal s with the analysing wavelet g is the convolution of s with a scaled and 
conjugated wavelet g a (t) = g(—t/a)/a, where the overbar denotes complex conjugation : 

S(r,a) = g a * s(t) = ^=Js (^T~) s{t) dt (A ' 2) 

It should be remarked that one uses often the so-called L 1 -normalisation, with a factor 1/ am 
(A.l) and (A.2), instead of 1/ \fa, in order to enhance small scales, where the finer details lie. 
In the Fourier domain, the expression (A.2) takes the following form: 

S(r,a) = - 1 - f G(acv)S(tv)e icVT dcv, (A.3) 

where S and G are the Fourier transforms of the signal s and of the wavelet g, respectively. The 
equations (A.2) and (A.3) show clearly that the wavelet analysis is a time-frequency analysis, 
or, more properly, a time-scale analysis (the scale parameter a behaves as the inverse of a 
frequency). In particular, the relation (A.3) shows that the CWT of a signal s is a filter with a 
constant relative bandwidth Aco/co = const. 

Then a straightforward calculation shows that this transform conserves energy (in the sense 
of signal processing), that is, 



ff\S(T,a)\ 2 ^=c g £js(t)\ 2 dt. 



(A.4) 



Clearly we must require the wavelet g to satisfy the so-called admissibility condition, namely, 

c^27r/|G(a,)| 2 0j<oo. (A.5) 



Wavelet-based techniques in MRS 



189 





-10 10 -10 10 

Fig. 20. Two usual one-dimensional wavelets: (left) The Mexican hat or Marr wavelet; (right) 
The real part of the 1-D Morlet wavelet, for coq = 5.6. 



Eq.(A.4) means that the CWT is an isometry from the space of signals onto a closed subspace 
Hg of L 2 (B?^,dadT/ a 1 ), where IR^_ denotes the scale-position half-plane IR^_ = {(r,a),T <G 
R, a > 0}. Therefore, the CWT may be inverted on its range l~Lg by the adjoint map, and this 
gives an exact reconstruction formula: 



5(0=Cj 1 //g( T#a) (0S(T / fl)-^. 



(A.6) 



This formula may also be interpreted as an expansion of the signal into the wavelets gt T>a \, 

with (wavelet) coefficients S(r,a). 

A necessary (and almost sufficient) condition for admissibility is that the wavelet have no DC 

component: 

G(0)=0 ^^ fg(t)dt = 0. (A.7) 

This is in fact the admissibility condition that is used in practice. 

This transform is very general in the sense that there is one CWT for each choice of the 
analysing wavelet g. For each application, one should select an analysing wavelet adapted 
to the type of signal at hand. For instance, in order to detect and to characterize the singular- 
ities of a signal or a curve, it is advantageous to use as analysing wavelet a derivative of the 
Gaussian, for instance, the familiar Mexican hat (Figure 20, left), 



g H (x) = (l-x z ) 



x 2 )e 



x 2 /2 



«=> 



G H (co) 



2 - 

cv e 



(A.8) 



In our case, MRS signals are relatively well defined in frequency, so it is more interesting to 
use analysing wavelets which are well localized in frequency space. This is the case of the 
Morlet wavelet, defined by 



g M (t) = e^e-^/^+hit) <* G M (cv) = Vtoi(r e 



-(cv-cv y 



°o /2 + H{cv), (A.9) 



where the correction term h is necessary to enforce the admissibility condition (in the sequel 
we shall use the value ctq = 1). If cooCq is sufficiently large (typically cooCq > 5.5), then h is 
numerically negligible, and will indeed be omitted. The Morlet wavelet can be interpreted as 
a bandpass linear filter centered around co = coq/ a and of weight l/faa) (Figure 20, right). 
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Fig. 21. Support properties of the Morlet wavelet g M : for a = 0.5,1,2 (left to right), g( Ta ) 
has width 3, 6, 12, respectively (top), while Gr TfU \ has width 3, 1.5, 0.75, and peaks at 12, 6, 3 
(bottom). 



All the results presented here have been obtained with the Morlet wavelet, but they can easily 
be generalized to any analysing wavelet whose Fourier transform has a single maximum at 
co = coq, or even to the Short Time Fourier Transform (STFT) 7 (Delprat et al., 1992). 
An important fact is the so-called reproduction property. Indeed it may be shown that the 
orthogonal projection Pg from L 2 (R.^_,dadT/a 2 ) onto the closed subspace Tig (the space of 
wavelet transforms) is an integral operator, with kernel 



K{T',a';T,a) = c g 1 {g {Tl>a , ) \g {T/a) ). 



(A.10) 



In other words, a function / £ ^(R^^adr/a 2 ) is the WT of some signal if and only if it 
satisfies the reproduction identity 



f{T',a') = JJK{T',a';r,a)f{T,a, 



dadr 



(A.11) 



For this reason, K is called the reproducing kernel of g. It is also the autocorrelation function g 
and as such it plays an essential role in calibrating the CWT (Antoine, 1994). 
Now the relation (A. 11) shows that the CWT is enormously redundant (the signal has been 
unfolded from one variable t to two variables (r,a)). Thus it is not surprising that the whole 
information is already contained in a small subset of the values of S(r,a). An example of 
such a subset is the so-called skeleton, that is, the set of ridges, which are essentially the lines of 
maxima of the modulus of the WT (in the case of a monochromatic signal, the ridges become 
horizontal lines a = a r , as we have seen in Section 2). Another example is obtained by taking 
an appropriate discrete subset T = {aj, t^} of the half-plane R 2 ^ , as it is necessary in any case 



7 The STFT is obtained by replacing scaling by modulation in the definition of the wavelets, that is, 
replacing Eq.(A.l) by g {r>a) (t) = e it/a g(t - t). 
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for numerical evaluation of the integrals. However, for most wavelets g, the resulting family 
{gf a . fTk \} is never an orthogonal basis (for the Morlet wavelet, for instance, the kernel K is a 
Gaussian, thus it never vanishes). At best, it is an overcomplete set of vectors, technically 
called a frame, provided I contains sufficiently many points (Daubechies, 1992). 

A.2. Localization properties and interpretation 

The main virtues of the CWT follow from the support properties of g. Assume g and G to 
be as well localized as possible (compatible with the Fourier uncertainty principle). More 
specifically, assume that g has an 'essential' support of width L, centered around 0, while G 
has an essential support of width Q, centered around coq. Then the transformed wavelets 
g( Ta ) and Gr Ta \ have, respectively, an essential support of width ah around r and an essential 
support of width CI/ a around coq/u. This behavior is illustrated in Figure 21, which shows 
the Morlet wavelet in the time and frequency domains, for three successive scales a = 0.5, 1 
and 2, from left to right. Notice that the product of the two widths is constant (we know it has 
to be bounded below by a fixed constant, by the (Fourier) uncertainty principle). Remember 
that 1/a behaves like a frequency. Therefore: 

• if a >> l,gf T/ a) is a wide window, whereas G( T/fl ) is very peaked around a small fre- 
quency cvq /a: this transform is most sensitive to low frequencies. 

• if a <C hg( T/ a) is a narrow window and G< T/a \ is wide and centered around a high 
frequency cvq/u: this wavelet has a good localization capability in the space domain 
and is mostly sensitive to high frequencies. 

Combining now these localization properties with the zero mean condition and the fact that 
g( Tf a) acts like- a filter (convolution), we see that the CWT performs a local filtering, both in time 
and in scale. The WT S(r, a) is nonnegligible only when the wavelet g( Tl a) matches the signal 
s(t), that is, it filters the part of the signal, if any, that lives around the time r and the scale a. 
Taking all these properties together, one is naturally led to the interpretation of the CWT as 
a mathematical microscope, with optics g, position t and global magnification 1/a. In addition, 
the analysis works at constant relative bandwidth (Acv/cv = constant), so that it has a better 
resolution at high frequency, i.e., small scales. This property makes it an ideal tool for detect- 
ing singularities (for instance, discontinuities in the signal or one of its derivatives), and also 
scale dependent features, in particular, for analysing fractals. 

A.3. Implementation questions 

Faced with this new tool, one must begin by learning the rules of the trade, that is, one must 
learn how to read and understand a CWT (Grossmann et al., 1990). The simplest way is to 
get some practice on very simple academic signals, such as a simple discontinuity in time or a 
monochromatic signal (pure sinusoid). We note that it is natural to use a logarithmic scale for 
the scale parameter a. The visual effect is that the lines, r/a = constant, are not straight lines, 
but hyperbolic curves; at the same time, the horizon a = recedes to infinity (see Figure 22). 
The analysing wavelet g is supposed to be complex, so that we may treat separately the mod- 
ulus and the phase of the transform. The scale axis, in units of In a, points downward, so that 
high frequencies (small a) correspond to the top of the plots, and low frequencies (large a) 
to the bottom. The results are presented by coding the height of the function by density of 
points (12 levels of gray, from white to black). The phase is 27i-periodic. When it reaches 2rc, 
it is wrapped around to the value 0. Thus the lines of constant phase with value Tkn are lines 
of discontinuity, where the density of points drops abruptly from 1 (black) to (white). In 
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Fig. 22. Morlet WT of a 5 function: (left) modulus; (right) phase. 



addition, the functions plotted are thresholded at 1% of the maximum value of the modulus 
of S(t, a). We will now analyse the two academic signals mentioned above, 
(i) A simple discontinuity 



The simplest signal is a simple discontinuity in time, at t 
The WT is obtained immediately and reads 



to, modelled by s(t) = 5{t — to). 



S(r / a) = a- 1/2 g(a- 1 (t -r)). 
The following features may be read off Eq.(A.12): 



(A.12) 



• The phase of S(t, a) is constant on the lines r/a = constant, originating from the point 
t = to on the horizon. These lines point towards the position of the singularity, like a 
finger. 

• On the same lines of constant phase, the modulus of S(t, a) increases as a~ 1 ^ 2 when 
a —> 0, so that the singularity is enhanced. The effect is even more pronounced if one 
uses the L 1 normalisation. 

This is illustrated on Figure 22, which presents the modulus and phase of the WT of a 5 func- 
tion, using a standard Morlet wavelet (but the result is independent of the choice of g). 
The interesting point is that this behavior is extremely robust. For instance, the 'finger 7 point- 
ing to a ^-singularity remains clearly visible when the latter is superposed on a continuous 
signal (even if the amplitude of the 5 function is too small to be invisible on the signal it- 
self), or even in the presence of substantial background noise. Similarly, the discontinuity 
corresponding to the abrupt onset of a signal is readily identified with the CWT. We refer to 
(Grossmann et al., 1990) for several spectacular examples. 

This is the origin of the edge or boundary effects that we have encountered in Section 4.1. The 
first notion is that of cone of confidence or cone of influence. Let the wavelet g vanish outside 
the interval Ig = [t m { n , t max ] . Then, given a point to in the support of the signal, the region 
in which it influences the WT is the cone r G al g + to = [— fl£ m in + to,at max + to]. Thus 
the region of influence increases linearly with a. The effect is clearly seen in Figure 1: the 
cones of influence of the two endpoints of the spectrum are the regions where the phase of the 
WT differs from that of a pure sinusoid (see (ii) below). This is the region to be avoided, as 
discussed in Section 4.1. 
(ii) A single monochromatic wave 
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Fig. 23. Morlet WT of a single sinusoid: (left) modulus; (right) phase. 



Equally simple is a single harmonic signal (monochromatic wave): 

1 



which gives 



s(t) 



S{t,o) 



Ju> s t 



& S(co) 



5{CjO-CjO s ), 



In 



G(acv s )e i(VsT = S(a,0) e il 



(A.13) 



(A.14) 



The same relations remain true for a real monochromatic signal, s(t) = sino; s f or s(t) = 

cos oo s t, if the wavelet g is progressive (that is, G(co) = for cv ^ 0). 

Again two important properties may be read off immediately from Eq.(A.14): 

• The modulus of S(r,a) is independent of r. Hence, the graph of |S(t,«)| consists of 
horizontal bands and the profile for a fixed time t essentially reproduces the profile of 
G. 

• The phase of S(r, a) is linear in r. Since the phase is 27i-periodic, the graph of <E>(r, a) := 
arg S(t, a) is a linear sawtooth function: 



0(r,a) = (jO s t (mod 2rc). 



(A.15) 



These properties are illustrated on Figure 23 for a single sine function analysed with a Morlet 
wavelet. This pattern of equidistant vertical black-to-white bands is the signature of a pure 
frequency signal. This can be seen already in Figure 1. 

Both the modulus and the phase allow to determine the frequency co s of the signal. If the 
modulus of the wavelet G(cv) has a single maximum for to = coq, Eq.(A.14) gives immediately 
co s = coq I a r , where a r is the scale corresponding to the maximum in the profile of |S(t, a) \ for 
fixed r. For instance, the (truncated) Morlet wavelet g(t) = exp(icoQt) exp(— 1 2 /2) yields: 



S(r,a) = Vae- 1 ^-^ 2 e i(VsT , 
and the result is obvious. As for the phase, Eq.(A.15) gives, at least locally: 






OJQ 
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(A.16) 



(A.17) 
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A.4. The discrete wavelet transform 

Notice that the discretized CWT which is used in practice, including in the present text, is 
totally different from the so-called discrete WT (DWT). Indeed, orthogonal bases of wavelets 
may be constructed, but from a completely different approach based on the notion of multires- 
olution analysis. 

We emphasize that the DWT is totally different in spirit from the CWT, either truly continuous 
or discretized, and they have complementary ranges of applications: 

. In the CWT, there is a lot of freedom in choosing the wavelet g, but one does not get an 
orthonormal basis, at best a frame. This is a tool for analysis and feature determination, 
as in MRS or other problems where the scaling properties of the signal are unknown a 
priori, for instance in fractal analysis. 

. In the DWT, one insists on having an orthonormal basis, but the wavelet is derived from 
the multiresolution analysis. This is the preferred tool for data compression and signal 
synthesis, and the most popular in the signal processing community. 

More radically, one may even say that the kind of problems treated here can be solved only 
with the CWT, the DWT is simply not adapted to the underlying physics, although it has been 
proposed for MRS (Neue, 1996). For instance, the algorithm for detecting spectral lines, as 
well as the ridge concept, rest upon a stationary phase argument. Similarly, the determination 
of fractal exponents exploits the scaling behaviour of homogeneous functions or distributions 
and the covariance properties of the CWT. All these notions are foreign to the DWT, which is 
more a signal processing tool. 
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1. Introduction 

According to the development of the Internet, multi-media contents such as music, picture, 
movie, etc. are treated by digital format on the network. It enables us to purchase digital 
contents via a net easily. However, it causes several problems such as violation of ownership 
and illegal distribution of the copy. Digital fingerprinting is used to trace back the illegal users, 
where unique ID known as digital fingerprints is embedded into digital contents before distri- 
bution Wu et al. (2004). When a suspicious copy is found, the owner can identify illegal users 
by extracting the fingerprint. The fingerprinting techniques of multimedia contents involve 
the generation of a fingerprint, the embedding operation, and the realization of traceability 
from redistributed copies. The research on such fingerprinting techniques is classified into 
two studies; secure cryptographic protocol and design of collusion resistant fingerprint. 
In a cryptographic protocol, the goal is to achieve the asymmetric property between a buyer 
and a seller such that only the former can obtain a uniquely fingerprinted copy because of 
the threat of dispute. If both of the parties know the fingerprinted copy, the buyer may re- 
distribute a pirated copy but later repudiate it by insisting that it came from the seller. An 
asymmetric protocol Pfitzmann & Schunter (1996) is executed by exploiting the homomor- 
phic property of the public key cryptosystem that enables a seller to produce the ciphertext of 
fingerprinted copy by operating an encrypted fingerprint with encrypted contents. 
Since each user purchases multimedia contents involving his own fingerprint, each copy is 
slightly different. A coalition of users will therefore combine their different marked copies 
of a same content for the purpose of removing /changing the original fingerprint. A number 
of works on designing fingerprints that are resistant against the collusion attack have been 
proposed. Many of them can be categorized into two approaches. One is to exploit the Spread 
Spectrum (SS) technique Cox et al. (1997); Wang et al. (2004; 2005); Zhao et al. (2005), and 
the other approach is to devise an exclusive code, known as collusion-secure code Boneh 
& Shaw (1998); Staddon et al. (2001); Tardos (2003); Trappe et al. (2003); Yacobi (2001); Zhu 
et al. (2005), which has traceability of colluders. Although cryptographic protocols provide 
the asymmetric property, the production of embedding information is based on the design of 
collusion-resistant fingerprint. 

In this chapter, we introduce the implementation method of watermarking technique in the 
encrypted domain during the fingerprinting protocol. As the robustness against attacks, a 
transformed domain like frequency domain is generally suitable to embed watermark infor- 
mation into an image. In such a case, the components of the transformed domain may be 
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represented by real values. In order to apply a public-key cryptosystem, all frequency compo- 
nents of an image must be quantized to integer. In the operation, a fingerprinting information 
is embedded to the quantized value. From the perceptual property, the changes in low fre- 
quency components stand out compared with that of the other components and hence each 
component is quantized adaptively by a special quantization step size. In the conventional 
method Kuribayashi & Tanaka (2005), for the embedding of an information bit of which value 
is unknown, the frequency components in the embedding positions are quantized to a special 
number before embedding so that the value can be changed depending on the information 
bit, which embedding method is based on QIM watermarking Chen & Wornel (2001). We pro- 
pose the method for implementing the spread spectrum watermarking technique by carefully 
designing parameters for rounding operation. As the precision of the representing watermark 
signal is sensitive for the implementation, the parameters are scaled by multiplying a constant 
factor. For the characteristic of the fingerprinting protocol, frequency components and the wa- 
termark signal must be separately encrypted after quantization. In such a case, the consistency 
of the precision is a sensitive issue. Then, the separate rounding operation causes interference 
term in a deciphered data at a buyer side. Without loss of secrecy of an original content, the 
interference term is removed after decryption in the post-processing. The proposed approach 
provides a guideline for the selection of watermarking technique suitable for a multimedia 
forensic system. 

2. Fingerprinting Protocol 

One of serious threats in the fingerprinting is dispute and repudiation of a purchase. The pur- 
pose of fingerprinting protocol is to solve such threats by achieving the asymmetric property, 
where only a buyer knows a fingerprinted copy. If both a buyer and a seller know a finger- 
printed copy, the seller cannot prove to a third party whose copy it was even if the buyer's 
fingerprint can be extracted. This is because a malicious seller may distribute the copy in order 
to frame an innocent buyer. Hence, it is desirable that only a buyer is able to obtain his own 
fingerprinted copy in the protocol. Such a protocol is called the asymmetric fingerprinting 
protocol. As in real-life market places, it is desired that electronic market places offer privacy 
to the customers. It should be possible to buy different articles anonymously, since purchased 
items can reveal a lot of behavioristic information about a buyer. The solution is the anony- 
mous fingerprinting protocol. Thus, the fingerprinting protocol is classified into the following 
three classes. 

Symmetric: The operation to embed a fingerprint is performed only by a seller. Therefore, he 
cannot convince any third party of the traitor's treachery even if he has found out the 
identity of a traitor in an illegal copy. 

Asymmetric: Fingerprinting is an interactive protocol between a buyer and a seller. After the 
sale, only the buyer obtains the copy with a fingerprint. If the seller finds the finger- 
printed copy somewhere, he can identify the traitor and convince a third party that the 
the copy is illegally distributed by the traitor. 

Anonymous: A buyer can purchase a fingerprinted copy without informing his identity to a 
seller, but he can identify the traitor later. It also retains the asymmetric property. 

In asymmetric fingerprinting, the plain value of a fingerprint should not be revealed to a seller, 
otherwise he can produce a fingerprinted copy by himself. Therefore an interactive protocol is 
performed to prevent the seller obtaining the fingerprinted copy. Such a protocol is based on 
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public-key cryptosystems because they assure only a buyer can decrypt a ciphertext though 
both of them can perform the enciphering operation. In order to achieve the asymmetric 
fingerprinting, a homomorphic property of public-key cryptosystems is applied. 

2.1 Asymmetric Property 

In order to achieve an asymmetric property, a homomorphic property of public-key cryptosys- 
tems is introduced in the fingerprinting protocols Pfitzmann & Sadeghi (1999). The homomor- 
phic property enables a seller to obtain the ciphertext of fingerprinted copy by operating an 
encrypted fingerprint with an encrypted original content. Since the ciphertext is computed 
using a buyer's encryption key, only the buyer can decrypt it; hence, only he can obtain the 
fingerprinted copy. 

The homomorphic property of public-key cryptosystems is often applied for cryptographic 
protocol as operations that can be performed without revealing the plain value. If an opera- 
tion on a ciphertext space results in an operation on the message space, the cryptosystem is 
homomorphic, and principally the former operation is multiplication and the latter is one of 
three operations, "addition, multiplication, exclusive or" , in public-key cryptosystems. 
Let E(M) be a ciphertext of a message M. The homomorphic property satisfies the following 
equation: 

g(E(Mi),E(M 2 )) = E(/(Mi,M 2 )), (1) 

where g(-) and /(•) is one of the operations, addition, multiplication, XOR, etc., which is re- 
lated to the applied cryptosystem and the embedding algorithm (Most public-key cryptosys- 
tems select multiplication for #(•)). If Mi is regarded as a digital content and M 2 as a fin- 
gerprint, the fingerprint can be embedded in the content without decryption by multiplying 
those ciphertexts. Since they are calculated using buyer's public encryption-key, the finger- 
printed copy is decrypted only by the buyer, hence the asymmetric property is satisfied. The 
embedding operation based on the homomorphic property is basically performed for each 
element of fingerprint information which will be composed of bit-sequence or spread spec- 
trum sequence, hence each element is separately embedded in its corresponding position. 
Thus, Mi is not the entire content, but one of the components like the frequency elements 
to be fingerprinted by a watermarking technique. Note that in watermarking techniques 
Katzenbeisser & Petitcolas (2000) for digital images, it is advisable to embed information in 
the frequency components for both the robustness and perceptual quality. When the vector 
representation of Mi is given by {m\ t \,m\^,m\^, . . .}, the ciphertext is also represented as 
E(Mi) = {E(mi / i) / E(mi / 2),E(mi / 3) / . . .}. As the consequence, the detail of Eq.(l) is given by 

g(E(m u ),E(m 2ii )) = E(f{m u ,m v )), (i = 1,2,3,...). (2) 

The multiplicative property of RSA scheme Rivest et al. (1978) is applied to embed a finger- 
print in Memon & Wong (2001), the homomorphism of a bit commitment scheme based on the 
quadratic residues Brassard et al. (1988) is exploited Pfitzmann & Sadeghi (1999; 2000), and 
the additive homomorphic property of public-key cryptosystem such as Okamoto-Uchiyama 
encryption scheme Okamoto & Uchiyama (1998) and Paillier cryptosystem Paillier (1999) is 
utilized in Kuribayashi & Tanaka (2005). In these schemes, to convince a seller that a trans- 
mitted ciphertexts really contains his fingerprinting information, zero-knowledge interactive 
protocol (ZKIP) must be performed, which is easily constructed using the applied public-key 
cryptosystem. Such characteristic is necessary for the security reason and the anonymity of a 
buyer is achieved. 
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Fig. 1. The flow of the asymmetric fingerprinting protocol. 



2.2 Asymmetric Fingerprinting Protocol Based on Bit Commitments 

In the asymmetric fingerprinting scheme, a buyer and a seller jointly embed a fingerprint. 
First, the buyer encrypts a fingerprint and sends it to the seller. Then the seller verifies that 
the received ciphertext is made from the real fingerprint, and embeds it in his encrypted copy 
by multiplying those ciphertexts. Finally, the buyer receives the encrypted and fingerprinted 
copy and decrypts it. After the protocol, only the buyer gets the fingerprinted copy without 
disclosing his identity. The model of asymmetric fingerprinting protocol is described in Fig.l. 
A concept of an anonymous fingerprinting protocol was first presented in Pfitzmann & Waid- 
ner (1997), and the fingerprinting system composed of several protocols was presented Pfitz- 
mann & Sadeghi (1999), which security was further improved in Pfitzmann & Sadeghi (2000). 
There are three parties, buyer B, seller S, and registration center 1ZC. First, 1ZC generates a 
pair of keys, secret key and public key, and distributes the latter to all participants of the sys- 
tem. When B begins a trade to a seller S, first B must register at 1ZC. And then B withdraws 
a digital coin which includes an identify proof W = proof (id) of his identity (fingerprint), id, 
and its signature which can be verified using the IZC's public key and can assure the legiti- 
macy of the buyer. In Fingerprinting Protocol, B encrypts his fingerprint and sends to S. Then 
using a zero-knowledge proof, B proves that the contents of the ciphertext is equivalent to 
that of W. After S is convinced the validity of the ciphertext, he encrypts his image, and mul- 
tiplies the received ciphertext and the ciphertext of his image to embed the fingerprint in his 
image based on a homomorphic property. In order to prove that the ciphertext really includes 
the fingerprint without revealing the plain value, two kinds of bit commitment schemes are 
applied. One is based on the discrete logarithm assumption, and the other is on the quadratic 
residues Brassard et al. (1988) which security depends on the p-subgroup assumption and 
quadratic residues assumption, respectively. The commitment schemes BCjji and BCrq are 
described as follows. 

BCjji: Let p be a large prime, and g and h be the generators. The commitment compi (b, r) of 
a bit b is calculated using a random number r as follows. 



comjji(b,r) = g h r (mod p) 



(3) 
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BCq R : Let p and q be large primes, and n = pq. The commitment coniQ R (b,r) is obtained by 
the following equation. 

com QR (b,r) = (-l)V (mod n) (4) 

Here, it is remarkable that the committed value b of BCjji is not only binary, it can take 
an integer of (Z/pZ). When W is calculated based on BCpi, namely W = comjji(id,r) = 
g id h r mod p, then it is difficult for a seller to embed directly the value of id using the commit- 
ment. Because of the characteristic of the commitment scheme, the recovery of the committed 
value is generally impossible. So instead of W, the commitment of each information bit of 
id = Yu w p-K which is calculated by comQ R (wj,tj), is applied for embedding. For a certain 
bit Xf e {0, 1} of digital contents, S computes the commitment corag^ (X z -, r z -), and multiplies 
coniQji(wj / rj) to it. 

comQ R (Xi,ri) • com QR (wj,rj) = (-l) Xf ^'(r z -r ; -) 2 (mod n) (5) 

It is noticed that if XjWj is 0, the result is quadratic residue, otherwise, it is quadratic non- 
residue. The knowledge of two primes p and q allow B to compute the value XjWj mod 2 
using the Jacobi Symbol while S can not determine that it is quadratic residue or not. So the 
security is based on the difficulty of factoring n = pq. 

Before the above computation, B must certify that the values comQ R (iVj,rj) of the commit- 
ments are equivalent to that of W. Using BCpi, B convinces S by zero-knowledge interactive 
protocol that the committed value of comjj^(id,r) is equivalent to that of comQ R (id,r). After 
the above protocol, only B can decrypt the fingerprinted copy and S can obtain the proof of 
the communication which can be used later if B illegally redistributes the copy. 
The function /(•) in the homomorphic property of BCq R is exclusive or operation. Based on 
the property, an encrypted fingerprint can be embedded in the encrypted copy, but the en- 
ciphering rate is extremely small because the commitment can contain only one-bit message 
in log 2 n-bit ciphertext, where n is composed of two large primes such that the bit-length of n 
should be more than 1024. Therefore, the enciphering rate of this method is more than 1/1024. 

2.3 Unbinding Problem 

It is also desirable for the fingerprinting protocol to solve the unbinding problem such that the 
relation between fingerprint information and a specific transaction performed by a buyer and 
a seller. In the elementary fingerprinting protocol Memon & Wong (2001), fingerprint infor- 
mation to be embedded is not well considered, which is merely related to user's information 
such as name, address, phone number, e-mail address, etc.. When a seller finds an illegal copy 
and detects the corresponding buyer by extracting the fingerprint, he will go to court with 
the collected proofs. A malicious seller, however, frames the detected buyer by embedding 
the obtained fingerprint into the other contents which are more expensive than the detected 
one what he really sold to the buyer. Therefore, once a seller obtains such a fingerprint, it 
is possible for him to transplant it into another much expensive contents so that he can get 
compensated more. 

In Lei et al. (2004), a fingerprint is binded with a common agreement (ARG) by producing the 
signature of a trusted watermark certification authority (WCA), and the transaction of digital 
contents is uniquely associated with a log file. For anonymity of buyers, a digital certifica- 
tion authority {CA) is introduced in the fingerprinting protocol. A buyer B first randomly 
selects a key pair (p/c#,sfcg), where pkg and skg are the public and secret keys of public-key 
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cryptosystem, respectively He sends pkg, which is a pseudonym associated with B, to CA in 
order to get an anonymous certificate CertcAiP^s)- When B makes an order to a seller S, he 
checks the validity of CertcA(p^s)- Then S asks WCA to generate a unique watermark W for 
the current transaction between B and S. The protocol between the buyer B and seller S is 
summarized below (the detail is referred to Lei et al. (2004)). 

1. B selects one-time key pair (pk*, sk*) and generates its certificate Cert p ] Cj3 (pk*) using the 
public key pkg. After making a common agreement ARG, B calculates a digital signa- 
ture Sign p ] c *(ARG) using the one-time public key pk*. B sends pk$, pk*, Certc^pks)' 
Cert pkB (pk*), ARG, and Sign pk * (ARG) to S. 

2. If the validity of the received items is verified, S generates a watermark V and embeds 
into contents X. The watermark is reference information to retrieve this sale record 
from illegally distributed copy; hence it could be omitted if the seller wants to avoid 
the degradation of quality Then, S send Cert p ] CB (pk*), ARG, Sign pk* (ARG), and X^ y ) 
to WCA. 

3. Upon receiving the items, WCA verifies the validity of the certificate and sig- 
nature, and reject the transaction if any of them is invalid. Otherwise, using 
x( y ) it generates a unique and robust watermark W as fingerprint information 
which is specific to this transaction. Then, it computes E pk *(W), E p ] CyvcA (W), and 
Sig n WCA(Epk* (W), pk*, Sign pk * (ARG)), and sends them back to S. 

4. When S receives the response, the embedding operation in encrypted domain is per- 
formed by computing 

%* (X {W ' V) ) = B P * (X W ) © Epic* ( W), (6) 

where implies the embedding operation based on the homomorphic property. Then, 
S delivers E pk *(x( W ' V )) to B. 

5. After decrypting the received E pk * (x( W/V ) ), B obtains the watermarked copy x( W/V ) . 

Where E pk ( • ) is an enciphering function using a public key pk. The flow of the transaction is 
summarized in Fig.2. 

The signature Sign^cAi^vk* (W), pk*, Sign p ^ (ARG)) explicitly binds W and ARG, which, in 
turn, uniquely specifies a particular digital content X, so it is impossible for S to transplant 
the watermark from an illegal copy to other contents. 

3. Asymmetric Fingerprinting Protocol Based on Additive Homomorphism 

The idea of the protocol Kuribayashi & Tanaka (2005) is to exploit the public-key cryptosys- 
tem with additive homomorphic property such as the Okamoto-Uchiyama encryption scheme 
Okamoto & Uchiyama (1998) and Paillier cryptosystem Paillier (1999) for anonymous finger- 
printing. 

3.1 Public-Key Cryptosystem with Additive Homomorphism 

After Goldwasser-Micali's scheme Goldwasser & Micali (1984) based on quadratic residuosity, 
Benaloh's homomorphic encryption function, originally designed for electronic voting and 
relying on prime residuosity, prefigured the first attempt to exploit the plain resources of this 
theory. Okamoto and Uchiyama significantly extended the enciphering rate by investigating 
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Fig. 2. The transaction of the fingerprinting protocol. 



two different approaches: residuosity of smooth degree in Z* and residuosity of prime degree 
p in Z* 2 , respectively. Here, we review the cryptosystem and enumerate the properties of the 
enciphering function. 

Let p and q be two large primes (\p\ = \q\ = £ p bits) and N = p 2 q. Choose g £r (Z/NZ) 
randomly such that the order of gp = g?~ x mod p 1 is p, where g.c.d.(p, q — 1) = 1 and 
g.c.d.(q, p — 1) = 1. Let h = g N mod N. Here a public key pk is (N, g, h, £ p ) and a secret 
key sk is {p, q). The cryptosystem, based on the exponentiation modN, is constructed as fol- 
lows. 

Encryption: Let m (0 < m < 2^ _1 ) be a plaintext. Selecting a random number r £# (Z/NZ), 
a ciphertext is given by 

(7) 



Decryption: Calculate first C p 



where 



C = g m h r (modN). 
CP -1 mod p 2 and then 

_ L(C P ) 



Hs v ) 



L(x) 



(mod p), 



(8) 



(9) 



We denote the encryption function E pk (m, r) and decryption function D sk (C). Three impor- 
tant properties of the scheme are given by the following PI, P2 and P3. 

PI. It has an additive homomorphic property : if m\ + m 2 < p, 



Epk(mi, r i) ' E pk (m 2/ r 2 ) = E vk (m x + m 2 , n + r 2 ) (mod N). 



(10) 



P2. It is semantically secure if the following assumption, i.e. p-subgroup assumption, is true: 
Epk(®' r ) — ^ r mod N and E pk (l, r') = gh r mod N is computationally indistinguish- 
able, where r and r' are uniformly and independently selected from £# (Z/NZ). 

P3. Anyone can change a ciphertext, C = E vk (m, r), into another ciphertext, C ! = Ch r mod 
N, while preserving the plaintext of C (i.e., C' = E pk (m, r")), and the relationship be- 
tween C and C' can be concealed. 
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B : buyer 

id = E Wj2 J 
a, € B , (Z/NZ) 

corrij = g w 'h a ' (mod N) 



< 9 g k h k (mod iV) * 



1- 



^fc(^) 



/» + Twj 



(mod AT) 



5 : seller 

W = g ld (mod N) 
a e R (Z/NZ) 
V = h a (mod N) 

n com/- 7 ' = W ■ V (mod JV) 
It, b t € B , (Z/NZ) 

Embedding Intensity J 1 



Yi 



g T ifl b i . comj 



(mod N) 



Fig. 3. Fingerprinting protocol based on additive homomorphism. 



Although the enciphering rate of Paillier cryptosystem Paillier (1999), which has the similar 
structure to Okamoto-Uchiyama encryption scheme, is higher, it requires more computations. 
So the selection of the scheme is dependent on the applied system. For convenience, the 
cryptosystem in the protocol is represented by Okamoto-Uchiyama encryption scheme; the 
approach can be easily translated to the Paillier cryptosystem, the readers are recommended 
to check the original paper Paillier (1999). 

3.2 Main Protocol 

The fingerprinting protocol is executed between a buyer B and a seller S. B commits his 
identity(fingerprint), id = J^WjlJ (0 < j < I — 1) to S the enciphered form, corny, where the 
values of Wj are binary. Then, S encrypts his image X z - (0 < i < L) and multiplies it to the 
received corny. We assume that B has already registered at a center 1ZC, and sent S the coin 
which includes a fingerprint and its signature. For simplicity, W = g ld mod N is regarded as 
a commitment of id. Under the assumption, the fingerprinting protocol is given as follows 
(indicated in Fig.3). 

[ Fingerprinting Protocol ] 

Step 1. S generates a random number a(2 £ < a < N) and sends it to B. 

Step 2. B decomposes a into £ random numbers a, £# (Z/NZ) to satisfy the following equa- 
tion. 

£-1 

a = £ ap) (11) 

j=o 
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Where the values of a\ to a^_\ are selected randomly under the condition, 



7=1 
and uq is calculated as follows. 



-l 

E a i lj < a > ( 12 ) 



£-1 
a = a~Y^ ap- ] (13) 

7=1 



A bit commitment of each Wj is calculated as 

com ] = g w Jh a J (mod N), (14) 

= E v k(wj,a.j) (mod N), (15) 

and sent to S. 
Step 3. To verify the commitment, S calculates 

V = h a (mod N), (16) 

and makes sure that the following equation can be satisfied. 

Yl com? = W-V (mod N) (17) 

i 

Step 4. <S generates L random numbers fcj £# (Z/NZ) and embedding intensity T of even 
number. Then, in order to get the encrypted and fingerprinted image, S calculates 

f g Xi h bi • comj (mod N) marking position 
Y t = Y . ; (18) 

[ g^h® 1 (mod N) elsewhere 

and sends it to B 

Step 5. Since the received Y z is rewritten as 

giXi+Tw^Taj+bi ( mod N j marking position 
g Xi h bi (mod N) elsewhere, 

B can decrypt Y z - to get the plaintext. 



(19) 



[ X z - + Twy (mod /?) marking position 
[ X z - (mod p) elsewhere 

On the deciphered message, if Wj = 1, then X z has been increased, and if Wj = 0, then nothing 
has done to Xv. 
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Remark 1: If we regard Wj as a message and a; as a random number, then corrij is represented 

by E pk (wj, dj) and comj by E pk (Twj, Taj) because 

comj = (g w ih a i) T (modN) 
= g T ™Jh Ta J (mod N) 
= E pk (Twj,Taj). (21) 

In many watermarking schemes, the embedding procedure is performed by an addition of wa- 
termark signal, namely a watermark is added to or subtracted from pixel values or frequency 
components with a certain intensity Therefore, the additive homomorphism is suitable for 
such watermark schemes. In Eq.(18), g Xi h bi = E pk (Xj, bj) is regarded as S's enciphered im- 
age, and then from the property PI Y z - at the marking position is rewritten as 

*i = Epk(Xi, h) ' Epk(Twj, Taj) 

= E pk (Xi + Tw jf Taj + bi) (22) 

If S uses Xj as a pixel value directly, the above operation can be applied easily Considering 
about the robustness against attack such as lossy compression and filtering operation, etc., the 
transformed domain is generally more resilience for such attacks. 

In the fingerprinting protocol B may be able to forge his identity as he has not proved that the 
values of Wj (0 < j < £ — 1) are binary. Even if they are not binary, Eq.(17) can be satisfied 
choosing them suitably. Then a malicious buyer may try to find the embedding position by 
setting the values adaptively. To solve the problem, a zero-knowledge interactive protocol 
has been introduced to prove that a commitment contains binary value, the procedure, called 
binary proof, is clearly described in Kuribayashi & Tanaka (2005). 

3.3 Modified Fingerprinting Protocol 

We consider the size of the message being encrypted, where the bit length of a message is 
revealed as the public key £p of Okamoto-Uchiyama encryption scheme. Since X z - and T are 
much smaller than 2^ _1 (< p) and the ciphertext is three times as large as p, the enciphering 
rate is still low. To exploit the message space effectively, the size of message to be encrypted 
should be modified as large as 2^ _1 . 
Let mi be 

J Xj + Twj marking position 

1 Xj elsewhere, 

and £ m be the maximum bit-length of m z -. Since £ m is much smaller than £p, the message can 
be replaced by 

7-1 

Mj/ = J^ m i/l+t 2 imt , < i < L/y - 1, (24) 



f=0 



where 



(25) 



It is illustrated in Fig.4. If the ciphertext of the message M? is calculated by S using comj and 
Xj in the fingerprinting protocol, the enciphering rate becomes at most 1/3 in theory. 
In order to perform the above operations, the fingerprinting protocol of Step 4 and Step 5 
presented in the fingerprinting protocol is changed as follows. 
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WV+l) 7 -2 


m (^+i) 7 -l 


^ i m bits^" C bits^ £ m bits^ 




^ C bits ^ C bits x 



M' : 



Fig. 4. Composition of the message M z / . 



[ Modified Fingerprinting Protocol ] 

Step 4. In order to get the encrypted and fingerprinted image y z -, S calculates 






g Xi - comj (mod N) marking position 



yX{ 



(mod N) 



elsewhere. 



(26) 



To synthesize some y z - in one ciphertext Y z v, the following operation is performed using 
a random number by e# (Z/NZ). 



Yi>=(n(yiiy+tf"")- hhi ' ( modN ) 



(27) 



Step 5. B decrypts the received Y,-/ to obtain M,/. Since he knows the bit-length £ m of m;, he 
can decompose M,v into the pieces, and finally he can get the fingerprinted image. 

Remark 3: From Eqs.(23)-(26) and the property P3, Eq.(27) is expressed by 

Y i' = (T\g mi '-> + ' 21 '"') ■ h r (modN) 
t 

= g^ m i'y^ tmt h r ( m odN) 

= g M ''h r (mod N) 
= E pk (Mi,,r). 



(28) 



If the Okamoto-Uchiyama encryption scheme is secure and the bit-length of My is less than 
£p, B can decrypt Yy = E(My,r). Here, in Eqs.(27) and (28) several pieces my 7 + t of finger- 
printed image that compose My are encrypted in one ciphertext E(My,r) / though each piece 
is encrypted in the original scheme. Therefore, My should retain a special data structure de- 
scribed by Eq.(24). If S changes the data structure, B can not decompose it into the correct 
pieces ntyy+t' and then he can claim the fact. Hence, with the knowledge of data structure B 
can decompose the decrypted message My into my 7 + t , and finally get the fingerprinted im- 
age. Furthermore, as Mjt is simply produced by composing several pieces of my 7 + t , B can not 
derive any information about original image from the decrypted message. 
Assume that the size of fingerprint is £ bits, and the fingerprint is embedded in the frequency 
components of an image where the number of components is L and each component is ex- 
pressed by £jn bits. Then the total amount of plain data of digital contents is £ m L. In Pfitz- 
mann & Sadeghi (1999) and Pfitzmann & Sadeghi (2000), the modulus n is a composite of two 
large primes. Since only one bit is encrypted when bit commitment schemes are used, each 
bit of the frequency components must be encrypted, thus the total amount of encrypted data 
is £ W L log 2 n bits. On the other hand, the modulus of the fingerprinting protocol with addi- 
tive homomorphism is N(= p 2 q, 3£p bits). In the original scheme, the amount of encrypted 
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conventional 


original 


modified 


l/3ip 


Ifjf/ Olp 


1/3 



Table 1. Enciphering rate. 



data is Llog 2 N(= 3£pL) bits as each component is encrypted. In the modified scheme, it is 
(Llog 2 N)/j (~ 3£ m L) bits, because from Eq.(25) there are at most L/j messages M z v to be 
encrypted, since £ m ~ £ m . Here, if log 2 n ~ log 2 N = 3£p, the enciphering rates are indicated 
in Table 1. Since the enciphering rate of Paillier cryptosystem is 1/2, the protocol can achieve 
the rate if the cryptosystem is applied instead of Okamoto-Uchiyama encryption scheme. 



4. Collusion Resilience 

In a fingerprinting scheme, each watermarked copy is slightly different, hence, malicious 
users will collect their copies in order to remove /alter the watermark. For an improperly 
designed fingerprint, it is possible to gather a small coalition of colluders and sufficiently at- 
tenuate each of colluders' fingerprint to produce a pirated copy with no detectable traces. 
Thus, it is important to model and analyze collusion, and to design fingerprints that can resist 
the collusion attack. 

There are several types of collusion attacks that may be used against fingerprinting system. 
One method is to average fingerprinted copies, which is an example of the linear collusion at- 
tack. Another collusion attack involves users cutting out portions of each fingerprinted copy 
and pasting them together to form a pirated copy. Other attacks may employ nonlinear oper- 
ations, such as taking the maximum or median of signal values of individual copies. As the 
countermeasure of collusion attack, a number of works on designing fingerprints have been 
proposed. One approach generates mutually independent sequences, e.g. spread spectrum 
sequence, for assigning users as their fingerprints, the other approach encodes fingerprint 
information considering the distances among fingerprint codes. 

On the former approach, spread spectrum sequences which follow a normal distribution are 
assigned to users as fingerprints. The origin of the spread spectrum watermarking scheme 
is Cox's method Cox et al. (1997) that embeds the sequence into frequency components of 
digital image and detects it using a correlator. Since normally distributed values allow the 
theoretical and statistical analysis of the method, modeling of a variety of attacks have been 
studied. Studies in Zhao et al. (2005) have shown that a number of nonlinear collusions such 
as interleaving attack can be well approximated by averaging collusion plus additive noise. 
So far, many variants of the spread spectrum watermarking scheme are based on the Cox's 
method. 

Let W be a watermark signal composed of £ elements Wj £ N(0, 1), (0 < i < £) and each 
of them is embedded into selected DCT coefficient X z -, (0 < i < £) based on the following 
equation, 

X z w = X i {l + ccw i ), (29) 

where N(0, 1) is a normal distribution with mean and variance 1, and cc is an embedding 
strength. At the detector side, we determine which SS sequence is present in a test image by 
evaluating the similarity of sequences. From the suspicious copy, a sequence W is detected by 
calculating the difference of the original image, and its similarity with W is obtained as fol- 
lows. 
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W • W 
sim(W,W) = , (30) 

vw -w 

If the similarity value exceeds a threshold, the embedded sequence is regarded as W. 
At the detection, DCT coefficients of test image are subtracted from those of original image, 
and then the correlations with every candidates of watermark signal are computed. Thus, 
non-blind and informed watermarking scheme can be applied. In fingerprinting techniques, 
the original content may be available at a detection because a seller is assumed as the author, or 
a sales agent who knows it. A simple, yet effective collusion attack is to average some variants 
of copy because when c copies are averaged, the similarity value calculated by Eq.(30) results 
in shrinking by a factor of c, which will be roughly VJ/c Cox et al. (1997). Even in this case, 
we can detect the embedded watermark and identify the colluders by using an appropriately 
designed threshold. 

Chen et al. Chen & Wornel (2001) showed that additive spread spectrum watermarking, in 
general, not good choices for embedding a bit-sequence, and, as an alternative, they intro- 
duced a new class of embedding strategies, which is referred to as " quantization index mod- 
ulation (QIM)". In the study, they presented that dither modulation is a practical implemen- 
tation of QIM that exhibits many of the attractive performance properties of QIM. The conve- 
nient structure of dither modulation, which is easily combined with error-correction coding, 
allows the system designer to achieve different rate distortion-robustness trade-offs by tuning 
parameters such as the quantization step size. It is also suitable for fingerprinting system by 
encoding fingerprint information by collusion-secure code. Thus, the combination of the QIM 
watermarking and collusion-secure code can provide a good fingerprinting system. 
Aiming at the extraction of a fingerprint bit-sequence, the QIM watermarking is implemented 
in Kuribayashi & Tanaka (2005) and its variants are employed in Prins et al. (2007). In Swami- 
nathan et al. (2006), the capability of the QIM based fingerprinting system is investigated, 
and the results show that one variant, which is called the spread transform dither modula- 
tion (STDM), retains an advantage under blind detection. Under non-blind detection, which 
is a reasonable assumption in fingerprinting system, there is still a performance gap with the 
spread spectrum method. It is noted that, in Yacobi (2001), the traceability is further improved 
by combining a spread spectrum embedding like Cox's method. 

Assume that the bit-length of the message space is £m and that of each watermarked frequency 
components is £ m . Generally, £m is much larger than £ m . In order to exploit the message 
space effectively, dozens of watermarked frequency components are packed in one message 
in Kuribayashi & Tanaka (2005), hence, the enciphering rate is almost equivalent to that of 
an applied cryptosystem by suitably designing the message space of a ciphertext. From the 
viewpoint of enciphering rate, the modification of QIM method implemented in Prins et al. 
(2007) is not a good choice, and the improvement of the robustness against attacks is still 
inferior to the spread spectrum method. The adaption of fingerprinting code further restricts 
the scalability of the QIM based fingerprinting system because of the long code-length. 

5. How to Implement Spread Spectrum Watermarking on Encrypted Domain 

Despite the simple structure of the QIM watermarking, the exploitation of fingerprinting code 
prevents the usability for various kinds of digital contents. We note that one major drawback 
of the conventional methods Kuribayashi & Tanaka (2005) Prins et al. (2007) is the long code- 
length of the fingerprinting code. Alternatively, the spread spectrum watermarking technique 
Cox et al. (1997) is implemented on the fingerprinting protocol based on the homomorphic 
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property of public-key cryptosystem in this section. Hereafter, for simplicity, the embedding 
of the reference information V, which is introduced in Lei et al. (2004), and a random number 
used for the encryption are omitted in the protocol. 

The embedding operation in Eq.(29) can be easily performed using the additive homomor- 
phic property of public-key cryptosystems such as Okamoto-Uchiyama encryption scheme 
Okamoto & Uchiyama (1998) and Paillier cryptosystem Paillier (1999). Remember that Eq.(22) 
is composed of two operations; multiplication and addition for g(-) and /(•), respectively. 
Since the multiplication is realized by the iteration of addition, the embedding operation is 
represented by the multiplication and exponentiation. Suppose that an original image is com- 
posed of L pixels and is represented by the DCT selected coefficients X z -, (0 < i < £) and the 
remain ones X if (£ < i < L), and a watermark signal is represented by Wj, (0 < i < £). Then, 
the embedding operation of Eq.(29) is executed in the encrypted domain as follows. 

E pk (Xi(l + OLWi)) = E pk (X { ) • E vk {wi)^ (31) 

The above operation can be directly applied for the operation in Eq.(6). Here, it is noticed 
that a watermark signal and DCT coefficients are generally represented by real value and they 
must be rounded to integer before the encryption. If such parameters are directly rounded to 
the nearest integers, it may result in the loss of information. Hence, they should be scaled be- 
fore rounding-off . In addition, a negative number should be avoided considering the property 
of a cryptosystem because it is represented by much longer bit-sequence under the finite field 
of applied cryptosystem, which affects the other packed ones described in Eq.(27). Hence, a 
rounding operation that maps real value into positive integer is required. 
At first, we show the operation concerning to a watermark signal W = {wq, w\, wi, • • • , we-i}- 
Since the ciphertext of W is computed by a watermark certification authority WCA, the en- 
ciphering operation is performed previously sent to a seller S. A constant value p w is added 
to each element of watermark signal Wj, (0 < i < £) to make the value positive. Then, it is 
scaled by a factor of s w in order to keep the degree of precision, and it is quantized to Wj. Such 
operations are formalized by the following one equation; 

w t = int(s w (wi + p w )), 0<i<£ (32) 

where int(a) outputs the nearest integer from a real value a. After the operation, WCA 
encrypts W = {wq, W\,wi, • • -,W£-i} using a public key pk, and the ciphertexts E pk (W) = 
{E pk ((WQ),E vk (wi),E pk {w2), • • '/Epk(™£-l)}' Pw' an d Sw are sent to S. It is noted that E p ] c (W) 
corresponds to E pk *(W) in Fig.2, and the corresponding ciphertext of E vkwcA (]N) is also sent 
to S. 

Next, S performs the rounding operation to DCT coefficients X if (0 < i < £) as follows. A 
constant value p x is added to each DCT coefficient, and then scaled by s w s x . By quantizing it, 
the rounded DCT coefficient X z is obtained. 

X { = int(s w s x (Xi + p x )), 0<i<£ (33) 

For the control of rounding operation of each DCT coefficient, the watermark strength oc is 
modified to ccf, 

~ol { = int(s x oc\Xi\), 0<i<£ (34) 

Using the above items, S embeds W[ into X z - for < i < £ based on the additive homomorphic 
property of public cryptosystem as follows. 

E pk(Xi) • E^WiY 1 = E pk (Xi + ociWi) (35) 
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Since the plain value of the ciphertext E pk (Xi + ~OL(UO{) is 

Xi + KiWi = s w s x (X i + p x )+s x 0c\X i \s w (w i + p w ), (36) 

= s w s x ((X i + ccw i \X i \) + (p x + cc\X i \p w )) / (37) 

the scaling factor s = s w s x and the adjustment factor p = p x -\- oc\Xi\p w are necessary to calcu- 
late the actual watermarked DCT coefficients X z - + ocwi\Xi\. Therefore, these two parameters s 
and p are sent to B as well as E pk (Xi + a z w z ). It is noticed that the remained DCT coefficients 
Xj, (£ < i < L) should be sent to B. In order to keep the secrecy of the embedding position, 
they must be encrypted before delivery Without loss of generality, the rounding operation for 
those coefficients are given by 

X { = int(s x s w (Xi + p x + oc\Xi\p w )), £ < i < L, (38) 

and the ciphertexts E^(X Z -) are sent with Ep k (Xj + ctjWj) to B. Namely, the ciphertexts of a 

watermarked image E pk (X ), which is corresponding to E pk * (x( W ' y ) ) in Fig.2, is composed 
of those ones. 

E ? k{X } = { E pk (X t ) £<i<L (39) 

After the decryption of the received ciphertexts E pk (X ), B divides the results by a factor 
of s, and then subtracts p as the post-processing operation. At the embedding position, the 
ciphertexts are E pk (Xj + ~oL(W{) and the post-processing operation outputs the fingerprinted 
coefficients X z - + ocWf \ X z - 1 as follows; 

sn V -^- -p = X t + ccWi\Xi\, 0<i<£, (40) 

where D sk (-) is a deciphering function using a secret key sk. At the other position, the cipher- 
texts are E pk (Xi) and B obtains X z - after the post-processing operation. 

Dsk{E f m -P = Xj,e<i<L. (41) 

It is remarkable that the embedding position is kept secret from B, the classification of the 
above operations is difficult. The diagram of the interactive protocol is shown in Fig.5. 
In Eq.(22), the watermarked coefficient X™ is composed of two terms; Xj and ocwjXi. Since 
W{ is encrypted at the center WCA prior to the embedding operation at S, X 7 - and W{ are 
rounded separately. Considering the post-processing at B, the scaling factors s Wr s X/ and the 
compensation factor p should be constant. Here, we assume that a constant value is uniformly 
added to real values which are Wi and X z to make it positive. Then, B must subtract the 
interference term related to both X z and Wj, which requires additional communication costs. 
If the adjustment factor p is varied with respect to X z -, the amount of information to be sent 
to B from S becomes very large. In order to avoid it, we set p a constant value by controlling 
the value p x . Even if p and oc is known, to obtain X z - is still informationally difficult because of 
three unknown parameters p x , p w , and X z - for a given one equation p = p x + oc\Xi\p w . As the 
consequence, the secrecy of the original DCT coefficients is assured. 

Notice that if the size of scaling factors s w and s x is increased, the proposed scheme can simu- 
late the original Cox's method more precisely. From the viewpoint of enciphering rate, how- 
ever, these factors should be small. Referring to the modified fingerprinting protocol, the 
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watermark Certification 
Authority WCA 

w l — > w l = int(s w (w l + p w )), < i < £ 

w l ->• E pk (wi) 7 <i < £ 

' E pk (Wi), p w , s w _ 



B : buyer 



5 : seller 



X -> X- = I int ( s ™ s *( x i + Px)) 

| int(s x s w (X l + p x + a\X % 

X % -> ^(XO, < i < L 
a — >■ c^ = mt(s z a|Xf|)j < z < £ 



<z <£ 
\p w )) £<i<L 



E pk (X z ) ■ E^Wi) * = E pk (X z + am), 0<i<£ 



E pk (X ), p, s' 



Xi + aWi\Xi\ 0<i<£ 
X { £<i<L 






<z <^ 
^<z < L 



Fig. 5. The procedure of fingerprinting protocol to embed the spread spectrum watermark. 



— Yj — 

bit-length of a watermarked coefficient X z - = X z - + 0L{Wi, which is represented by a constant 

bit-length £ Xf is much smaller than that of message space in cryptosystems such as Okamoto- 

— W 
Uchiyama encryption scheme and Paillier cryptosystem, and some of X z should be packed 

in one message M; 



M: 



xrnx, w 



i+ll 



IX. 



w 



!+£-!/ 



(42) 



where £ is the number of packed coefficients and is dependent on s w and s x . Such a packing 



w 



operation is easily performed by computing the £ x t-ih power of E pk (X i+t ); 



7 w 



f=0 



(43) 



The appropriate size of s w and s x are explored by implementing on a computer and evalu- 
ating the simulated performance. It is worth mentioning that the enciphering rate of Paillier 
cryptosystem approaches asymptotically 1 using the extension of the cryptosystem Damgard 
& Jurik (2001) and then more data can be packed in one ciphertext. Although the works in 
Fouque et al. (2003); Orlandi et al. (2007) can encode rational numbers by a limited precision, 
they are not suitable for the packing operation. 

6. Simulation Results 

Since the basic algorithm of our scheme is Cox's scheme with a limited precision, we evaluate 
the degradation of image quality by PSNR, and the detected correlation values compared with 
the original values. If the results are similar, we regard that the performance is not degraded. 
In our simulation, a standard gray-scaled image "lena" of 256 x 256 pixels is used. The length 
of watermark signal W is £ = 1000 and the embedding intensity is a: = 0.1. Even if p w and 
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p x are added, the values of W{ and Xj might be negative. In such a case, the values are simply 
rounded to 0. 

The comparison of PSNR and correlation values for the watermarked image which is not 
distorted by attacks are shown in Fig.6 and Fig.7, respectively The PSNR of original Cox's 
scheme is 34.93 [dB] and the correlation value is 31.91, which are drawn by dot line in the 
figures. From the figures, we can see that the performance is asymptotically reaching the 
original value according to the increase of the scaling factors s w and s x . As the basic algorithm 
is Cox's scheme with a limited precision, we can regard that the performance is not degraded 
when the detected correlation values are similar. 

One of the important characteristic in the spread spectrum watermarking technique is the 
orthogonality of each watermark signal because of the robustness against collusion attack. It 
is well-known that the original scheme retains the robustness with a dozen of colluders. Under 
averaging collusion with 5 users, the average similarity value of original scheme is 13.64, and 
the proposed one is shown in Fig. 8. The robustness against the combination of collusion attack 
and JPEG compression are compared, which results are shown in Fig.9. From the results, the 
degradation of performance from the original scheme is very slight, and it does not affect the 
robustness against attacks. It is noted that the scaling factors s w and s x are closely related 
to the degradation of performance. It is better to increase the value of these parameters, for 
example s w > 2 3 and s x > 2 3 , but we have to consider the communication costs because the 
bit-length to represent the watermarked DCT coefficient X z - + ct{Wj is increased according to 
the size of s w and s Xr which degrades the coding rate of such information. For other images, 
"aerial", "baboon", "barbala", "fl6", "girl", and "peppers", the similar results are derived 
with the above parameters as shown in Table 2 and 3. The attenuation of PSNR value from the 
original one is at most 0.1%, that of the correlation value is at most 0.3%, and under averaging 
collusion the attenuation is less than 1%. As the consequence, recommended parameters are 
s w = 2 3 and s x = 2 3 from the simulation results. 

— W 
When we use the above recommended parameters, the value of X z - can be represented by 

20 bits (the range must be within [0, 2 20 ] if s w = s x = 2 3 ). For the security reason, the bit- 
length of a composite n = pq for the modulus of Paillier cryptosystem should be no less than 
1024 bits. When \n\ = 1024, an 1024-bit message is encrypted to an 2048-bit ciphertext. Un- 
der the above condition, the number of watermarked DCT coefficients in one ciphertext is at 
most 51 (= [1024/20J). Since the number of DCT coefficients are 65536 = 256 x 256, the 
number of ciphertexts is 1286 (= [65536/51J ) and the total size of the ciphertexts is about 
2.5MB, which is about 40 times larger than the original file size 66KB. In case the packing is 
not performed, the total size is more than 128MB. Therefore, we can conclude that the pro- 
posed method efficiently implements the Cox's spread spectrum watermarking scheme in the 
asymmetric fingerprinting protocol. 

7. Conclusion 

In this chapter, we investigated an asymmetric fingerprinting protocol with additive homo- 
morphism and a method for implementing watermarking technique in an encrypted domain 
for assuring the asymmetric property of fingerprinting system. We developed the commit- 
ment scheme utilized to achieve the asymmetric property, and enhance the enciphering rate 
by applying Okamoto-Uchiyama encryption scheme for the cryptographic protocol that re- 
tains additive homomorphism. In order to contain information in one ciphertext as much as 
possible, the large message space is effectively partitioned by multiplexing each fingerprinted 
and encrypted component of an image. 
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Fig. 6. The image quality for the scaling 
values s w and s X/ where that of original 
scheme is 34.93 [dB] depicted by dot lines. 




Fig. 8. The average correlation value after 
averaging collusion attack for the scaling 
values s w and s x . 



Fig. 7. The correlation values for the scal- 
ing values s w and s x , where that of original 
scheme is 31.90 depicted by dot lines. 




Fig. 9. The average correlation value after 
averaging collusion attack and JPEG com- 
pression with quality 35% for the scaling 
values s w and s x , where the average value 
of original scheme is 10.10. 



We proposed a new of approaches for collaborating the proposed asymmetric fingerprinting 
protocol and watermarking technique. In the conventional implementation, the QIM water- 
marking is applied to the fingerprinting protocol exploiting the quantization procedure that 
truncates a real value to integer, which is unavoidable process to apply the public-key cryp- 
tosystem based on the algebraic property of integer. In the method, fingerprint information 
must be coded by a fingerprinting code to be robust against collusion attack. It also causes an- 
other issues such that the applicable contents are limited to huge contents like movie because 
of the long code-length. In this chapter, we implemented the spread spectrum watermarking 
to be applicable for various kinds of contents. After exploring the fundamental properties of 
signals in an encrypted domain, a fingerprint sequence is scaled up in order not to attenuate 
the signal energy by quantization. Moreover, the effects of rounding operation that maps a 
real value into a positive integer are formulated, and an auxiliary operation to obtain a water- 
marked image is presented. From our simulation results, the identification capability of our 
algorithm is quite similar to the original spread spectrum watermarking scheme, hence we 
can simulate the scheme on the cryptographic protocol with a limited precision. 



Recent Fingerprinting Techniques with Cryptographic Protocol 



215 





aerial 


baboon 


barbala 


fl6 


girl 


lena 


peppers 


original 


36.34 


34.96 


34.61 


35.59 


35.49 


34.96 


34.48 


proposed 


36.35 


34.95 


34.61 


35.59 


35.48 


34.95 


34.48 



Table 2. The degradation of the image quality when s u 







aerial 


baboon 


barbala 


fl6 


girl 


lena 


peppers 


No attack 


original 


31.91 


31.91 


31.91 


31.91 


31.87 


31.91 


31.91 


proposed 


31.87 


31.82 


31.85 


31.85 


31.79 


31.84 


31.85 


Collusion 


original 


13.66 


13.64 


13.65 


13.65 


13.54 


13.64 


13.65 


proposed 


13.61 


13.50 


13.54 


13.57 


13.40 


13.54 


13.55 


Collusion 
+ JPEG 35% 


original 


11.60 


9.14 


8.95 


9.74 


9.01 


10.10 


10.27 


proposed 


11.56 


9.18 


8.91 


9.73 


9.18 


10.06 


10.16 



Table 3. The degradation of the correlation values when Su 
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Abstract 

We address in this contribution a problem stemming from functional data analysis. Assuming 
that we dispose of a large number of shifted recorded curves with identical shape, the objec- 
tive is to estimate the time shifts as well as their distribution. Such an objective appears in 
several biological applications, for example in ECG signal processing. We are interested in the 
estimation of the distribution of elapsed durations between repetitive pulses, but wish to esti- 
mate it with a possibly low signal-to-noise ratio, or without any knowledge of the pulse shape. 
This problem is solved within a semiparametric framework, that is without any knowledge of 
the shape. We suggest an M-estimator leading to two different algorithms whose main steps 
are as follows: we split our dataset in blocks, on which the estimation of the shifts is done by 
minimizing a cost criterion, based on a functional of the periodogram. The estimated shifts 
are then plugged into a standard density estimator. Some theoretical insights are presented, 
which show that under mild assumptions the alignment can be done efficiently. Results are 
presented on simulations, as well as on real data for the alignment of ECG signals, and these 
algorithms are compared to the methods used by practitioners in this framework. It is shown 
in the results that the presented method outperforms the standard ones, thus leading to a 
more accurate estimation of the average heart pulse and of the distribution of elapsed times 
between heart pulses, even in the case of low Signal- to- Noise Ratio (SNR). 

1. Introduction 

1 .1 Description of the problem 

Due to the improvements of electronic apparatus and registration systems, it is more and more 
common place to collect sets of curves or other functional observations. Such registration 
is often followed by a processing operation, since they tend to represent the same repeated 
phenomenon. In this contribution the problem of curve registration and alignment from a 
semiparametric point of view is addressed. More specifically, we assume that we dispose of 
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M registered curves, each of which being described by the model given in Equation (1.1): 

Vm{t) = s(t - e m ) + cremit), m = . . . M (1.1) 

where s is a curve of interest, m is an unknown shift parameter distributed according to 
some probability density function fg, cr is a real positive number and e m is a standard Gaus- 
sian white noise process. We are interested in the estimation of the curve s and of the shifts 
{0 m ,m = . . . M} or, when the number of curves M is large, in the estimation of the shift 
distribution f e . Such a problem appears frequently in functional data analysis (FDA) applica- 
tions, and we refer to Silveman & Ramsay (2005) and Ferraty & Vieu (2006) for examples and 
case studies related to this issue. In the framework described by the latter equation, the knowl- 
edge of the translation parameter 6, and more specifically of its distribution, can be used to 
determine the inner variability of a given cluster of curves. Several papers (see Ramsay (1998), 
Ramsay & Li (1998), Ronn (2001), Gasser & Kneip (1995),Kneip & Gasser (1992)) focus on this 
specific model for many different applications in signal processing for biology. For example, 
in neuroscience, neurons emit randomly electrical pulses which are recorded by an electrode. 
Biologists, in many applications, are interested in the estimation of the inter-spike interval, 
that is, either the estimation of the durations of elapsed time between two electrical pulses, 
or the estimation of its distribution. As stated in Johnson (1996), it is interesting to model the 
observed electrical signal as the sample path of a renewal process. We can find in recent con- 
tributions (see Pouzat et al. (2004) and Delescluse & Pouzat (2006)) the usefulness of the ISI 
for spike sorting. In those applications, it is often easy to segment roughly the signal such that 
we retain only one pulse into each segment, however the realignment of the obtained curves 
are mainly based in either alignment of the main structural information of the curves (e.g. the 
zeros, as in Gasser & Kneip (1995); see Kneip & Gasser (1992) for a description of available 
tools to characterize curves structural information), either in the knowledge of the shape of a 
standard electrical pulse, as in Ramsay (1998) or Ramsay & Li (1998) (in that case, the problem 
is often called template matching, see Lewicki (1998) and references therein). However, both 
approaches are sensitive to the level of noise, and some recordings are sometimes too noisy 
to authorize a satisfactory realignment of the curves. We are therefore interested in finding a 
method of estimation robust enough in relation to the noise level. 

1.2 The curve alignment and estimation from a statistical point of view 

The problem of the estimation of the shift parameter 9 has been investigated in numerous sta- 
tistical publications, and this according to two different approaches. The main contributions 
is this topic focus on the case of a finite number of curves, and provide asymptotic results 
when the number of sample points tend to infinity. For example, Dalalyan et al. (2006) stud- 
ied the case of two shifted curves, and proposed a penalized Maximum Likelihood Estimator 
approach, whereas Gamboa et al. (2007) suggested a semiparametric joint estimation proce- 
dure in the case of / curves (/ being a fixed number). Some functional data analysis techniques 
have also been described in Ramsay (1998) and Ramsay & Li (1998), and the authors generally 
assume that the shift can be expressed as a warping function which has to be estimated. The 
methods described in Gasser & Kneip (1995) and Kneip & Gasser (1992) are based on tem- 
plate matching procedures; for example, the latter suggested to estimate the sets of the local 
maxima of s, and to align the different curves accordingly. It shall be noticed that template 
matching approaches give indeed good results when the curve s is regular enough and the 
noise variance cr is small; however, they fail when the common shape s shows higher variabil- 
ity or in the case of low SNR. In the case of a finite number of curves, Lavielle & Levy-Leduc 
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(2005) suggested a semiparametric approach for the estimation of the period of a laser signal, 
thus following the lead of Ritov (1989). 

Another way of looking at the same model has also been proposed: instead of fixing the 
number of observed curves, it is interesting to make this number tend to infinity and to look 
at the obtained asymptotics. The first paper dealing with the estimation with a large number 
of curves can be found in Ritov (1989), and has received a larger attention in the last years. 
For example, Castillo (2006) and Castillo & Loubes (2007) propose to relate to the nonlinear 
inverse problem methods and derive estimates based on the works of Dalalyan et al. (2006), 
whereas Ronn (2001) suggested a nonparametric maximum likelihood estimator approach. 
More recently, Bigot et al. (2008) and Bigot & Gadat (2008) investigated the estimation of the 
shape s for an identical model, and suggested a wavelet approach which leads to a near- 
optimal (in the minimax sense) estimator of the shape. However, their assumption is that the 
shift distribution fg is fully known, thus enabling a deconvolution step in order to compute a 
nonparametric estimator of s. 

1.3 Curve alignment for ECG data 

In this contribution we focus more specifically on the analysis of ECG signals. In recordings 
of the heart electrical activity, at each cycle of contraction and release of the heart muscle, we 
get a characteristic P-wave, which depicts the depolarization of the atria, followed by a QRS- 
complex stemming from the depolarization of the ventricles and a T-wave corresponding to 
the repolarization of the heart muscle. We refer to (Guyton & Hall, 1996, Chapter 12) for an 
in-depth description of the heart cycle. A typical ECG signal is shown in Figure 1. 
Different positions of the electrodes, transient conditions of the heart, as well as some malfunc- 
tions and several perturbations (baseline wander, power-line interference, additional elec- 
tromyographic signal)Fotiadis et al. (2006); Sarnmo & Laguna (2006), can alter the shape of 
the signal. We aim to situations in which the heart electrical activity remains regular enough 
in the sense that the shape of each cycle remains approximately repetitive, so that after prior 
segmentation of our recording, the above model still holds. This preliminary segmentation 
can be done, for example, by taking segments around the easily identified maxima of the QRS- 
complex, as it can be found in Gasser & Kneip (1995). It is therefore of interest to estimate the 
shift parameters m in (1.1). These estimates can be used afterwards for a more accurate esti- 
mation of the heart rate distribution. In normal cases, such estimation can be done accurately 
by using the common FDA method (e.g. by using only the above prior segmentations). How- 
ever, when the activity of the heart is more irregular, a more precise alignment can be helpful. 
This happens for example in cases of cardiac arrhythmia, whose identification can be easier 
if the heart cycles are accurately aligned. Among interpretations deduced from ECG data, 
some are based on the so called "signal averaged ECG" (SAECG). SAECG is routinely used in 
clinics for late potential detection, various heart diseases and arrhythmia detections, as men- 
tioned in Cain et al. (1996), and more specifically for ventricular tachycardia and late potential 
detection ( see e.g. Nava et al. (2000), Rodriguez et al. (2000)). Analysis of the SAECG sig- 
nal is usually performed by using standard wavelet decomposition Englund et al. (1998). As 
mentioned in the cited contributions, SAECG is simply a signal averaging technique used to 
improve the signal-to-noise ratio, since clinicians assume that the ECG waveform is invariant 
and that the background noise is uncorrelated. Moreover a timing reference (i.e. a landmark) 
is set at the peak R of the QRS complex (since it is easily detectable) allowing averaging with- 
out distortion. This method is to relate to Kneip & Gasser (1992), where the authors chose 
several landmarks instead of one. We argue that this is actually a crucial point since jitter of 



220 



Signal Processing 



x10 




100 200 300 400 500 600 700 800 

Fig. 1. Typical ECG signal of an healthy subject (arbitrary units). 
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this temporal reference would distort the resulting SAECG. The proposed method leads to an 
estimation of the mean cycle by averaging the segments after an alignment according to an 
estimated Oj. Additional benefits for a more proper alignment can be found in many other 
measurements done by cardiologists. 

1.4 Chapter organization 

This chapter is organized as followed: Section 2 describes the derivation of the shift estimators 
and the density estimate. Roughly speaking, the estimation of the shifts is based on an M- 
estimation procedure with a cost function connected to the spectral density of the signal, and 
we suggest a plug-in estimation for the shift density; This leads to two different algorithms for 
curve alignment which are described in the end of the section. Some theoretical aspects are 
described in Section 3. Eventually, Section 4 is dedicated to results on simulations (in order 
to compare the performances of the two shifts algorithms) and real datasets, and discusses 
the influence of the standard perturbations which appear in ECG signal processing in the 
described algorithms. 



2. Methodology 

We present in this section the main assumptions used in the rest of the chapter and the deriva- 
tion of the proposed estimator. 
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2.1 Main assumptions 

We assume that after getting an electrical signal such as the one described in Figure 1, a prelim- 
inary segmentation is done so that we relate only to the model described in the introduction. 
We can therefore assume that we have observed sampled noisy curves on a finite time interval 
[0, T] . Each of these curves has being shifted randomly by some random variable 0. A typical 
curve is expressed in 1.1 and the number of curves is denoted by M. Assuming that the pre- 
liminary segmentation has been done efficiently and that each segmented curve has a whole 
single repetition of the signal, the following assumptions can be done: 

(H-l) The distribution of the shift (denoted by f$) and the shape s both have finite sup- 
port, respectively [0, Tq] and [0, T s ]. We also assume that s ^ on [0, T s ] and that the 
derivative s f is bounded. 

(H-2) Tq + T s < T; that is, we assume that the shape is fully observed in the all the recorded 
curves. 

As mentioned in Ritov (1989), this is equivalent to consider observations on a circle and setting 
T = 2 7i. Therefore, T will be chosen equal to In without any loss of generality. We also assume 
that there is no dependency between the shifts and the additional noise, that is: 

(H-3) The random variables {0j, I = . . . M} are independent and identically distributed, 
and are in addition independent from {e/,/ = 0...M}. 

2.2 Description of the shift estimation procedure 

In this section, we present a method for the semiparametric curve alignment. This method 
can be used as a first step for a nonparametric estimation of the shift density, by following the 
methodology described in Castillo (2006): first provide an estimate for the shifts, using in their 
example the methodology of Dalalyan et al. (2006), and then plug the obtained values into a 
standard kernel estimate. We propose an M-estimator to retrieve the shifts, in which the shape 
information is considered as a nuisance parameter and the shifts are estimated jointly. A sim- 
ilar approach appeared for example in Vimond (2008) leading to another estimator with good 
asymptotic properties. In another contribution, Gamboa et al. (2007) proposed a semiparamet- 
ric method for the shifts, with applications to traffic forecasting. This M-estimate, based on a 
criterion function related to Fourier coefficients, has been shown to be consistent and asymp- 
totically normal as the number of Fourier coefficients increases. However, in this contribution, 
we chose to focus on the asymptotic properties as the number of curves increases (of course, 
we also assume that the number of sampling points is large enough, since the M-estimation 
might lead to inconsistent results otherwise). This approach is explained by the fact that we 
have in practice little control on the sampling frequency, whereas obtaining a larger dataset of 
curves is easier. 

Following the method of Castillo (2006), we propose to plug M estimates of shifts into a kernel 
estimate. Consequently, we need to estimate the sequence {0/, I = 0...M}. One important 
difference, compared to the previously cited works, is that we choose to estimate blocks of 
parameters jointly instead of one at a time. We therefore split our dataset of curves in N 
blocks of K + 1 curves each, as indicated in Figure 2. Observe that the curve yo is included 
in each block, since we wish to align each curve accordingly to yo ; consequently, it shall be 
assumed in the rest of the chapter that 0q = 0. The interest of splitting the dataset of curves 
into blocks is double: it reduces the variance of the estimators of the shifts by estimating them 
jointly, and also provides smooth cost functions for the optimization procedure detailed in this 
section. Indeed, since the recorded signals are based on the same curve s, the average of the 
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Power Spectral Densities is close to the Power Spectral Density of the average curve provided 
the shifts are known and have been corrected. This idea will be the cornerstone of the M- 
estimation procedure proposed in this contribution. We thus estimate jointly the sequence of 



A yo{t) A yo{t) 



VK-l(t) ,,. 

A ilk y(N-i)K+K-it f J 



f% y*(0 )% VNK{t) 



Block 1 

Fig. 2. Split of the curves dataset 

vectors {Om, m = 1 . . . N}, where for all integer m 
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(2.1) 



The estimation of {Om, m = 1 ... N} is achieved by minimizing N cost functions, which are 
now detailed. Let us denote by S y the squared modulus of the Fourier Transform of a given 
continuous curve y, that is for all w\ 



Sy(cv) 






h dt 



This quantity is of interest, since it remains invariant by shifting. For each integer m = 1 . . . N, 
we define the weighted mean of K curves translated by some correction terms ff m = 

( A: (m-l)K+l/---/<W): 
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(2.2) 



k=(m-l)K+l 



where A(X) is a positive number which depends on K, and is introduced in order to give more 
importance to the reference curve i/o- m the rest of the paper, we shall write A instead of \(K) 
in order to avoid cumbersome notations . We now consider the following function: 
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(2.3) 
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The function described in (2.3) represents the difference between the mean of the Power Spec- 
tral Densities and the Power Spectral Density of the mean curve. Observe that (2.3) tends to 
a constant if the curves used in (2.2) are well aligned, that is when cxm is close to 6m. Since 
the observed curves are sampled, we will approximate the integral of Sy by its Riemann sum, 
that is we shall use 

1 » r * 2 

A /-i \ J- Y — i / \ 2inmk 

s y( k ) = - L y(^) e n 

m=\ 

as an estimator of Sy) that is, the Discrete Fourier Transform (DFT) of a sampled curve 
{y(tm), m = 1 ... ft} will be used in practice instead of the actual Fourier transform of a 
curve y. The following cost criterion is then introduced and should be minimized in order to 
reshift all curves into the n-th block: 
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(2.5) 



The M-estimator of Om, denoted by m , is therefore given by 

m = Argmin CKmG[0;27r] K||C m (Q: m )||^ 

Remark 1. It can be noticed that all blocks ofK + 1 curves have one curve y§ in common. We chose 
to build the blocks of curves as described in order to address the -problem of identifiability. Without this 
precaution, replacing the solution of (2.5) by 6 + c + 2kn, k <G Z and s by s(- — c) would let the 
cost criterion invariant. Adding curve yo as a referential allows to estimate 6 — Oq, thus avoiding the 
non-identifiability of the model. 

In order to define the criterion function, we chose to split the set of observed curves in N blocks 
of K + 1 curves. Indeed, this is not useful if the spectral information is fully known. However, 
since we observe noisy curves, and since we did not assume any knowledge on the spectral 
information, the functions Sy have to be estimated. A well known nonparametric estimator is 
the periodogram, which has been extensively studied (see e.g. Chonavel (2000) and references 
therein). This estimator is known to be asymptotically unbiased, but its variance does not tend 
to in the general case; moreover, the pointwise estimation leads to uncorrelated estimated 
values, therefore the periodogram provides an estimate of the power spectral density of a 
process with many irregularities, regardless of the regularity of the true power spectrum. A 
good way to reduce the variance of this estimator is given by the averaged periodogram (or 
Bartletfs method), based on the mean of several periodogram estimators, thus the necessity 
of splitting the dataset. We refer to Chonavel (2000) for a more detailed description of this 
method. 

It is interesting to compare the cost function introduced in (2.5) to the estimator introduced 
in Gamboa et al. (2007). In their contribution, they introduce additional weights to smooth the 
constrast function, for a fixed number of curves /. More precisely, they propose to estimate 
{Qj, ] — 1 . . . /} the minimum of the following criterion function: 
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(2.6) 



where {Si, I £ Z} is real sequence such that YLi$? < °° an d Ez $f < °°/ an d &n is the /-th 
discrete Fourier coefficient associated to the ;-th curve. This is to relate to Welch's method to 
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reduce the variance of the periodogram, and makes sense since the number of curves in Gam- 
boa et al. (2007) is assumed to be fixed. The asymptotics is then provided as the number of 
samples per curve tends to infinity. Similarly, the method proposed in Castillo (2006) leads to 
similar smoothing, since the shifting step is done by re-aligning one curve at a time, without 
using the fact that in the case of shift density estimation, the number of curves is assumed to 
be big. We argue that the method of averaged periodogram may be preferred for simplicity to 
weighted periodogram for variance reduction, since we only have one parameter to tune. 

2.3 Description of the algorithmic procedures 

2.3.1 Curve alignment with respect to a reference curve 

Following the derivation of the M-estimator, the first algorithm for curve alignment is pretty 
straightforward: the dataset is split in blocks which include the reference curve and an op- 
timization procedure (e.g. a conjugate gradient descent) is then performed on each block. 
The reference curve is by default the first one. The standard plug-in estimate is then used to 
estimate the density. The procedure is summarized in Algorithm 1. 

Algorithm 1 Alignment procedure w.r.t. a reference curve (Al) 

Inputs: input curves yo, . . . ,i/m where yo is the reference curve and parameters K, A 
Outputs: Shift estimators - {§j,j = 1... M} 

1 M 
Compute the average of the curves periodograms S = — — - ^ Sy. . 

M + 1 ;=0 

Split the curve dataset into N blocks of K + 1 curves, each of them including yo. 
for m = 1 . . . N do 

A 1 / mK \ 

ym(t;a 7n ) = —— - Ay (0+ E 1// (* + **) 
K ^ A \ j=(m-l)K+l / 

Define C m (a m ) =5- Sy m (. ;am ) 

Compute 6 m = Argmin aG[0;2 ^ K ||C m (a m )||^ where 6 m = {^ m _ 1 ) K ^ 1 ,...,§ mK } 
end for 
Return {§j,j = 1 . . . M} 

Algorithm (Al) is somehow problematic, since it involves the choice of a reference curve yo 
and giving it more importance. This may lead to a bad estimation of the shifts between curves, 
in the case of very noisy curves (when a is for example of the same order of magnitude than 
the signal), since the noise is also magnified in the process. In order to address this issue, it is 
possible to use another algorithm described in 2.3.2. 

2.3.2 A two-stage algorithm for curve alignment and shift density estimation 

The two stage algorithm (denoted by (A2) in the rest of the chapter) can be described as fol- 
lows: the dataset is split into N blocks of K curves, and the reference curve yo is not included 
in the blocks. The algorithm primarily performs alignment of the curves within each block. 
An average of the aligned curves of each block is calculated after this first step. The set of 
average curves is eventually aligned by similar means. The final shift estimator of each curve 
is the sum of the shift estimator of the curve among the curves of its block and the shift of the 
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averaged curve of that block; thus with the two stage algorithm the reference curve is unnec- 
essary. This is the main advantage of a two-stage algorithm, since choosing a reference curve 
with possibly low SNR leads to significant alignment errors. 

The two-stage procedure (A2) is summarized in Algorithm 2. In the first stage, estimation of 
the vector of shifts as in (2.1) is done for each block separately. Since we discard the reference 
curve yo/ tne average of curves in blocks m G {1, . . . , N} translated by some correction terms 

A 



A 1 ' mK 



ffm = 0( w _i)x+i, • • • , oc mK ), is equal to 

K \k={m-\)K+\ 
A cost function is then defined as it was for algorithm (Al): 

A 1 M 



CmicXrn) = — — £ % - §y M (. ;am ) , (2.7) 

1V1 "•" 1 k=0 

and the first stage ends by the computation of the M-estimator which minimizes the cost 
function defined in (2.7). At this stage we obtain the shift estimates {0 m ,m = 1...N}. At the 
second stage an average of the aligned curves within each block is calculated, and we align 
the averaged curves. That is, the average curve is 

A 1 / mK \ 

ym{t) = ^[ E yrt + h) , 

\k=(m-l)K+l J 

and for a given correction term /3 = (jSi, . . . , fa) the mean of translated average curves is 

A 1 / N 

The second cost function for alignment between averaged curves is therefore 

A 1 M 

An M-estimator is now calculated for the shifts among blocks 

^ = Argmin /3G[0;27r]N ||C(^)||^. 

Finally the estimator for the shift of \j\ is the sum of the estimator obtained at the first stage of 
(A2) for this curve and the estimator obtained at the second stage for the averaged curve of 
the block \)\ belongs to, that is: 

(0 m ) k = d m + (d m ) k ,m = l...N,k = l...K. 
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Algorithm 2 Two-stage alignment procedure (A2) 

Inputs: input curves yi, . . . , \)m and size of each block K 

Outputs: Shift estimators - {§j,j = 1 . . . M} 

Split the curve dataset into N blocks of K curves. 

1 M 
Compute the average of the curves periodograms S = — ^ S y . 

E ^C + ^y) 
;=(m-l)K+l / 

Define C m (a m ) = 5 - Sy m( . ;am ) 

Compute W = Argmin aG[0;27r]x ||C w (a m )||^ where 6 m = {S( m _ 1 ) K + 1 ,...,8 mK } 




Compute y m (t) = - ^ y fc (f + 1 



end for 

A 



E 

yfc=(m-l)jK+l 

N 



Define C^) = 5 - S^..^ 
Compute £ = Argmin CKG[0;27r] K||C(^)||2 
for m = 1 . . . N do 
f or i '. = 1 ... X do 

(Qm)i — ^m + (Otnji 
end for 
end for 
Return 0\,...,0^. 



3. Theoretical properties 

We provide in this section some theoretical results about the cost functions obtained in the 
latter section. In order to prove the efficiency of the method, it is indeed necessary to check 
that all cost functions have global minima at the actual shifts. We also have to verify that, 
provided the number of curves is large enough, the cost functions are smooth enough so that 
the optimization can be done efficiently. The presented results hold for the algorithm (Al), 
though they can be easily extended to each stage of (A2). 

3.1 Expansion of a cost function 

Recall that the total number of curves is M = NK + 1, where N is the number of blocks and 
K is the number of curves in each block. The first curve yo is a common reference curve for all 
blocks. We denote by c s (k) the discrete Fourier transform (DFT) of s taken at point k, 

CsW 4l£ s(fm)e -^, 

n m=\ 
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and by fyj the DFT of \j\ taken at point k: 



1 n 

fk,l = ~ E^(^) e " 



Using this notation, relation (1.1) becomes in the Fourier domain for all k = — ^~^ . . . ^-^ and 
Z = 0...M: 

1 Yl 

fa = e ~ M ' Z E s ('« + e/)e"^ + -^ (V W + iWjy) 

" m=l V" 

= e^cft) + Odls'llcon" 1 ) + -^ (V w + iWjy) , (3.1) 

where in the latter equation e\ a constant such that \e\\ < nn~ x , and s f is the first deriva- 
tive of s which we assumed to be bounded. This error term which results from the sampling 
operation is purely deterministic, and is further on neglected since it is not expected to in- 
duce shift estimation errors greater than the length of a single bin (i.e. ft -1 ). The sequences 

{ V k/U k = - ^F 1 • • • ^ } and j W k/l , k = - ^ . . . ^ } are independent and identically dis- 
tributed with same standard multivariate normal distribution J\f n (0, I n ). We now compute the 
cost function C m associated with block m: 

n-l 
\\Cm(ct m )\\l = E ( A MW - BmikrOrr,)) 1 
k=0 
n-l 
+ £ (B m (k,6m) ~ BmiKcXrn)) 2 
k=0 
n-l 

+ 2£(B m (k,0 rn )-B m (k,a m ))(A M (k)-B m (k,0 rn )), (3.2) 

fc=0 

where A M (k) is the first term of the right hand side of (2.4) and B m (k,cx rri ) is the second 
term of the right hand side of (2.4), both taken at point k. Each term of the latter equation is 
expanded separately. We get that 

rr 2 M / \ 



1=0 

M 



(M- 

?rrhn(V (1c)) M 

~ LliTr L(YU sin (^) + W Kl cosCW/)) . (3.3) 

The last two terms of (3.3) converge almost surely to as M tends to infinity, according to 
Assumption (H-3) and the law of large numbers. Moreover, the sum of the second term is dis- 
tributed according to a x 2 distribution with M + 1 degrees of freedom. Thus, the term A^ik) 
tends to \c s (k) | + 2n~ l <j 2 as M — » oo. We now focus on the expansion of the terms associated 
to ||Ci(otx) !!§/ s i nce all other cost functions may be expanded in a similar manner up to a 



228 



Signal Processing 



change of index. The first curve of each block is the reference curve, which is considered to be 
invariant and thus has a known associated shift ocq = 6q = 0. We obtain 



Bi(fc,ai) 



1 



A + K 



A(c s « + — (y fc/0 + iW fc/0 )) 



+ £ (e*to-*)c 8 (k) + -^e 1 ^ (V k/l + iW M ) 



thus, if we define the sequence {A m , m = ... X} such that Aq = A and A m = 1 otherwise: 



Bl(fc,ai) 



l Cg WI v^ A , A e M*i-ei-otm+e m ) 
(A + ^) 2 £o A/Ame 
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E A z A w {e i 



k(oci-oc m ) s 



[V K iV Km + w Kl w Km + i(V kJ W Km - w kJ v Km )}} 

7=77 , ^ 2 1j A;A m e^' m '(V km -lW k „ 



act{k) 



V^(A + K)2 



I] A/A„ 



3 if:(e m +aj-a m ) 



(YJU+iWjy) 



(3.4) 



l,m=0 



We now can study the behavior of the cost function, as the number of curves tend to infinity. 
The functional ||Ci (oti) H2 can be split into a noise-free part, that is a term without random 
variables V or W, and a random noisy part. 

3.2 Decomposition of the cost function into a noise-free part and a noisy part 

Recall that the noise-free part of ||Ci(ai) H2 neither depends on <^ Vj^/, k = — rL ^- . . . ^^ |> nor 

\ Wfy, k = — ^^ . . . rL ^- >), and is denoted further by Di(ai). This term is equal to: 



D 1 (ai)=E|c fl (*)| 4 

fc=0 



1 K 

— r 2^ A ™ e 



X + A 



ra=0 



(3.5) 



Details of the calculations can be seen in Trigano et al. (2009). Note that due to (3.5), D\ 
has a unique global minimum which is attained when oc m = 6 m , for all m = 1 ... K, that is 
the actual shift value. We now provide two quantitative results about algorithm (Al). Their 
demonstrations can be also found in Trigano et al. (2009). 

Proposition 1. Let {rj(K,\), K > 0} be a sequence such that n(K,A)^0asK^ +00 for all A, 

and let ocbea real -positive number. Assume that for all k = . . . n — 1: 



1 



(K + A )^o 



]JH Ajexp(i/c(0 m -ctm)) 



> l-tj(K,\), 
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then there exists two positive constants j, and Kq such that, for K > K$, there is a constant c(K, A) 
such that the number of curves whose alignment error with respect to 6 — c(K,X) is bigger than 
rj (K,A) a , denoted by #{m = 1. . .K : \oc m — m — c(K,A)\ > n(K, A) 01 } is bounded as follows: 



#{m : \oc m -e m -c{K,A)\ >n(K,X) 0C } < y(K + A)n(K,A) 



l-2a 



Proposition 1 can be intuitively interpreted as follows: provided the optimization procedure 
is effective enough, if the number of curves in each block is large enough, most curves will 
tend to align, but not necessarily with respect to the reference curve t/q. Consequently, the 
weighting factor A is introduced in order to "force" all the curves in a block to align with 
respect to xjq, and the following proposition holds: 

Proposition 2. Assume that A is an integer, and that 

yn{K,X) l - lDi < 



K + X 

Then, under the assumption of Proposition 1, we get that \c(K, A) | <n(K, A) a 
In other words, when choosing A(X) such that 

\(K) ^+oo, ^^ ^OasX^oo, 
K. 

it is possible to obtain estimates very close to the actual shifts. In order to check that the 
optimization procedure can indeed be done effectively, we need the noisy part of the cost 
function to be small under the same conditions. 

We now study the noisy part of ||Ci(ai) H^. Recall that due to Equation (3.2), the noisy part 
of ||Ci (oil) ||| stems from terms of the form A M (k) — Bi{k,0\) and Bi{k,0\) — Bi{k,ot\). It is 
then possible to show the following proposition: 

Proposition 3. Assume that K — >• oo, A —> 00 and that A/K — »• 0, and let e be any positive num- 
ber. Let us denote by R(k) the noisy part associated to Bi{k f 0\) — Bi(k,cx\); we get under these 
assumptions that 

?rr 2 

A M (k) - B 1 (k,0 1 ) = — +0 F (n- 1 X" 1 / 2 ) 
n 

and 

R(k)=O w (n- 1 K- 1/2 + e )+O v (n- 1 ) 

Proposition 3 is of importance, since it shows that under the assumptions that each block 
contains a large number of curves, and that the weighting factor A is important but negligible 
with respect to K, the cost function || C\ (ai ) || 2 reduces to D\ (ai) plus a constant term, which 
means that the minimum of ||Ci(ai)|| 2 is with high probability close to the minimum of 
Di(ai). In practice, a typical choice of the weighting parameter would be A = K@ ', with 
< j6 < 1, and verifies all the assumptions of the previous propositions. 
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4. Applications 

We present in this section the results obtained by the presented algorithms, both for simu- 
lated and real data. In our simulations, the shape s is created according to a model used in 
neuroscience, namely the Hodgkin-Huxley model (see Johnson (1996) and references therein), 
and measure the performances of the shift alignment procedures by using the MISE (Mean 
Integrated Squared Error) criterion; more specifically, the shifts {6 m ,m = . . . M} are drawn 
accordingly to a known probability distribution, then estimated by means of the previously 
described algorithms and, as a measure of performance, the MISE of the obtained plug-in 
density estimates proposed in the end of Section 2 is computed. 

We first investigate the choice of the tuning parameters of the algorithm (Al), that the num- 
ber of curves in each block K and the weighting parameter A, and discuss the influence of 
these values in the obtained density estimator, for different choices of a 1 . We then investigate 
the performances of the algorithms (Al) and (A2), and how they compare to two standard 
methods: 

— The alignment method with respect to the local extrema, as proposed in Gasser & Kneip 
(1995) (we denote this algorithm by (A3), and 

— a measure of fit based on the squared distance between the average pulse and the shifted 
pulses leading, which is often used by practitioners and is described in Silveman & 
Ramsay (2005) (this algorithm is denoted by (A4). 

It shall be seen that from the point of view of the shift density estimation, the algorithm (Al) 
outperforms the others. 

We then apply the presented algorithms for the computation of the SAECG ; the real ECG 
data have been taken from the MIT-BIH database, and we present the obtained results for 
three different types of ECG recordings: 

— recordings stemming from a normal heart 

— recordings from a patient suffering of cardiac arrhythmia 

— recordings of noise-stress tests, that is with a lower SNR. 

4.1 Results on simulated data 

4.1.1 Experimental protocol 

Simulated data are created accordingly to the discrete model 1.1 and we compute the estima- 
tors for different values of the parameters K, A and a 2 . For each curve, we sample in order 
to get 512 points equally spaced on the interval [0;2tt]. We make the experiment with s sim- 
ulated according to the Hodgkin-Huxley model of a neural response. The shifts are drawn 
accordingly to a normal distribution A/'(0,32 2 ), and 6q = 0. 

4.1.2 Results 

We study the influence of the parameter K and A empirically by providing the MISE of the 
plug-in density estimates for different values of K, A and a 1 , with N = 100. In all these 
experiments, the value of the weighting parameter A is chosen accordingly to K, that is A = 
X 09 . The numerical values of the MISE are provided for all algorithms in Table 1. 
It shall be observed that the algorithm (Al) outperforms the other algorithms in most of the 
cases, except the case of extremely low SNR. We now present graphical results obtained by 
means of (Al), (A2), (A3) and (A4), in the case of high SNR, which can be obtained for example 
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K=10 


K=20 


K=30 


K=50 


K=100 


a 2 = 


(Al) 

(A2) 
(A3) 
(A4) 


0.0305 
0.0407 
0.0306 
0.0316 


0.0228 
0.0357 
0.0234 
0.0248 


0.0198 
0.0372 
0.0199 
0.0227 


0.0153 
0.0372 
0.0156 
0.0174 


0.0106 
0.0375 
0.0109 
0.0136 


a 2 = 10" 4 


(Al) 

(A2) 
(A3) 
(A4) 


0.0312 
0.0399 
0.0325 
0.0322 


0.0218 
0.0383 
0.0232 
0.0219 


0.0183 
0.0362 
0.0212 
0.0192 


0.0156 
0.0364 
0.0183 
0.0168 


0.0121 
0.0364 
0.0158 
0.0126 


a 2 = 10" 2 


(Al) 

(A2) 
(A3) 
(A4) 


0.0296 
0.0410 
0.0306 
0.0303 


0.0218 
0.0383 
0.0232 
0.0219 


0.0172 
0.0384 
0.0192 
0.0182 


0.0143 
0.0371 
0.0172 
0.0155 


0.0120 
0.0355 
0.0143 
0.0125 


a 2 = l 


(Al) 

(A2) 
(A3) 
(A4) 


0.0326 
0.0460 
0.0547 
0.0510 


0.0274 
0.0407 
0.0806 
0.0450 


0.0248 
0.0374 
0.0514 
0.0414 


0.0255 
0.0381 
0.0553 
0.0393 


0.0288 
0.0395 
0.0741 
0.0370 



Table 1. MISE of the density estimates obtained for different values of the noise variance a 2 
and number of curves in each block K, for a fixed value of A = X 09 



by choosing cr = 0.01, and in the case of low SNR, which can be obtained by fixing cr = 0.1. 
Two typical curves are presented in both cases in figure (3). 





(a) High SNR (a 2 = 10" 6 ) 
Fig. 3. Ten typical curves for different SNR values. 



(b) Low SNR (a 2 = 10" 2 ) 



Results are presented for the four algorithms and for these specific choices of cr in figure (4) 
and figure(5). 

Eventually, a graphical comparison of the actual shift distribution and its plug-in density esti- 
mate is presented in figure (6), and the results with a weighting parameter A too small is given 
in figure (7). 
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(a) Algorithm (Al) 



(b) Algorithm (A2) 





(c) Algorithm (A3) (d) Algorithm (A4) 

Fig. 4. Results of the curve alignment procedures for K = 30, N = 100, A = X 09 , o 2 = 10~ 6 . 



4.1.3 Discussion 

The graph obtained in figure (7) well illustrates Proposition 1. In this graph, we observe that 
in each separate block the curves are well aligned, since for each block plotting the actual 
values of the shifts versus their estimated values gives lines with slope 1. However, they do 
not align with respect to the location of the reference curve, due to a weighting parameter A 
too small. Taking a larger A allows to address this problem, as it can be seen in figure (4(a)) 
and figure (5(a)). From the obtained results of Table 1, it can be observed that the algorithm 
(Al) outperforms the others, from the point of view of density estimation. This algorithm is 
also robust to the noise level a, as it can be seen from figure (4(a)), figure (5(a)) and the results 
displayed in Trigano et al. (2008). 

Not surprisingly, the algorithm (A2) is well suited for alignment (e.g. in order to compute 
the average signal), but is less adapted for density estimation, since the curves do not align 
accordingly to the reference curve, as stated in Proposition 1. This means that, provided the 
expectation of fg is known, the algorithm (A2) would give results close to (Al). The two-step 
algorithm should be preferred for example in the case of very low SNR, where each curve is 
very noisy so that there is no good choice available for a reference curve x/q. Indeed, when 
comparing the results of (Al) and (A2) for a 2 = 0.01 and a 2 = 1, we can see that the MISE 
degradation is less important for (A2) than for (Al). 

The method described in Gasser & Kneip (1995), from which the algorithm (A3) is derived, is 
based on the features of the curves s, and is less effective when the noise level is important. 
This can be observed from figure (5(d)) and the results given in Table 1 for the highest val- 
ues of a 2 . This is predictable, since any method which relies on a preliminary smoothing of 
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(a) Algorithm (Al) 



(b) Algorithm (A2) 





(c) Algorithm (A3) (d) Algorithm (A4) 

Fig. 5. Results of the curve alignment procedures for K = 30, N = 100, A = X 09 , a 2 = 10~ 2 . 



the curves and an alignment with respect to the maxima of the curve would lead to a more 
important error than a semiparametric approach if the SNR is low. However, (A3) competes 
relatively well with (Al) in the case of high SNR, and may be preferred for simplicity in that 
case. Nevertheless, since (Al) performs an efficient alignment both for high and low values 
of a 2 , since this parameter introduces only a constant term in the cost functions C n , which can 
be omitted in the optimization procedure. We may argue that the main advantage of (Al) lies 
in its generality, and that it should be used for example when the SNR is unknown. 
Eventually, it can be observed that (A4), which relates to FDA methods described in Silveman 
& Ramsay (2005), is relatively efficient, provided the total number of curves is large enough. 
The MISE results as well as figures (4(c)) and (5(c)) indicate that this method is well fitted for 
curve alignment. However, a degradation of the performances can be noticed for the lowest 
values of K. This makes intuitively sense, since a small value of K indicates that only a few 
curves are used to compute the average signal at each iteration of (A4). Therefore, the align- 
ment w.r.t. the average signal yields a larger MISE. This appears e.g. in figure (4(c)), where a 
slight bias between the actual shifts and their estimators can be observed. Consequently, the 
algorithm (A4) gives results quite similar to (Al), but requires one block of N x K curves to 
compare well to (Al), which leads to significantly longer computational times. 
From the theoretical point of view, the good performances of (Al-2) with respect to (A3-4) can 
be explained by the study of another M-estimate proposed in Gamboa et al. (2007) for curve 
alignment, which gives further insight in the comparison with the state-of-the-art method. 
Indeed, (Gamboa et al., 2007, Theorem 2.1) shows that a statistically consistent alignment 
can be obtained only when filtering the curves and aligning the low-frequency information. 
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Fig. 6. Comparison between the true shift density (dotted red) and its plug-in estimate (blue). 



Therefore, an approach based on the spectral information is more susceptible to achieve good 
alignment by comparison to the method of Silveman & Ramsay (2005). 

4.2 Results on real ECG data 

We now wish to compare our method to the state-of-the-art for the alignment of heart cycles, in 
order to estimate the average signal. We provide the study of the signal presented in figure (1), 
which was obtained from the MIT-BIH database, and is a recorded signal stemming from both 
a healthy (figure (8) and (10)) and arrhythmic heart (figure (12)). 

4.2.1 Experimental protocol 

In order to obtain a series of heart cycles, we first make a preliminary segmentation using 
the method of Gasser & Kneip (1995), namely alignment according to the local maxima of 
the heart cycle. We then apply our method, and compare it to the alignment obtained by 
comparing the mean curve to a shifted curve one at a time. We took in this example K = 30 
and A = 10 for 3 iterations. 



4.2.2 Results 

We present on the following figures results obtained by algorithms (Al-4) on three different 
data sets from the MIT-BIH database. Figure (8) show results obtained on (very) noisy ECG. At 
first sight, all four methods seem to perform equally well regarding P and T waves. However, 
zoomed QRS patterns on figure (9) show that alignment procedures (Al) and (A2) give better 
results. Indeed, the black mean curve is smoother (jitter is reduced) than the one obtained 
with the two other methods, reflecting that alignment is achieved in a better way. On the 
contrary, mean curves obtained by algorithms (A3) and (A4) are less satisfactory since they 
are less temporally concentrated ; Moreover, the average curves obtained in that case show 
several local maxima, which is far from the standard shape of the QRS complex (indeed, we 
are expecting to find one single mode in this part of the signal, as given by the algorithms (Al) 
and (A2). 
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Fig. 7. Results of the shift estimation (Al) for K = 200, N = 30, A = 10, a 1 = 10 _1 . 



Figure (10) show results obtained with actual power-line interference and baseline shifting. 
On this data set, algorithm (A2) is outperformed by the 3 others as revealed by figure (11(b)). 
In this case, (A3) and (A4) perform well since maxima are. Note that (Al) mean curve is 
slightly smoother than (A3) and (A4) ones. 

Figure (12) and (13) show alignment of an arrhythmic data set. These data are typically en- 
countered in routine clinical use and contain ECG with very various waveforms and artifacts. 
(A3-A4) clearly achieve alignment according to maxima. Jitter is high in this case, since the 
shape of the curve can significantly vary from one pulse to another. Consequently, the mean 
curve does not reflect this variability. Such behavior from (A3-A4) could be expected, since 
one underlying assumption is that the curve shape s is made of peaked common noise-free 
patterns. By contrast, (A1-A2) achieve better alignment since the resulting jitter is compara- 
tively reduced and the corresponding mean curve smoother. The shape variability is intrinsi- 
cally taken into account by the way of periodogram coefficients. 

4.3 Discussion 

Results presented in the previous paragraph show clearly different behaviors of alignment 
procedure. Algorithms (Al) and (A2) fully take into account all frequencies composing curves 
including the different noise contributions, such as low frequencies representing baseline vari- 
ations or more localized ones such as power-line interference. By contrast, (A3) and (A4) are 
based on curve maxima meaning only high frequencies. When these maxima are perturbed 
by noise or distorted by intrinsic curve change (such as arrhythmic ECG), the corresponding 
mean curve does not reflects such perturbations. We believe that it makes our periodogram- 
based alignment methods more robust to the classical ECG perturbations. 

5. Conclusion 

In this contribution several methods of curve alignment for repeated events were introduced 
and investigated; this led to plug-in estimate of the density of elapsed times between events. 
Their performances were presented both on simulations and real ECG, and compared to the 
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Fig. 8. Results of the curve alignment procedures on real ECG for K = 30, N = 10, A = X 09 . 



well-known shift correction methods based on the alignment with respect to the local extrema 
(which is the standard method used in the ECG signal processing framework). It is shown 
that the algorithms which are based on a semiparametric approach outperform the methods 
based on an FDA approach. The suggested algorithms provides excellent results, whether 
the SNR is high or low. On real ECG data, the proposed algorithms gives good results for 
the computations of the SAECG, and seem according to the experiments extremely robust to 
distortions such as power-line interference, baseline wander or variability of the pulse shapes. 
Further results on the rates of convergences of the density estimator and the applicability of 
the method for the detection of heart diseases should appear in future contributions. 
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(a) Algorithm (Al) 





(b) Algorithm (A2) 




(c) Algorithm (A3) (d) Algorithm (A4) 

Fig. 9. Zoom on figure (8) on each aligned QRS region. 




(a) Algorithm (Al) 





(b) Algorithm (A2) 




(c) Algorithm (A3) (d) Algorithm (A4) 

Fig. 10. Results of the curve alignment procedures on real ECG for K = 30, N = 10, A = K - 9 . 
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(c) Algorithm (A3) (d) Algorithm (A4) 

Fig. 11. Zoom of figure (10) on each aligned QRS region. 




(a) Algorithm (Al) 




(c) Algorithm (A3) 




(d) Algorithm (A4) 



Fig. 12. Results of the curve alignment procedures on real arrhythmic ECG for K = 30, N = 10, 
A = K - 9 . 
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(c) Algorithm (A3) 
Fig. 13. Zoom on figure (12) on each aligned QRS region. 



(d) Algorithm (A4) 
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1. Introduction 

Intra-only video coding is a widely used coding method in professional and surveillance video 
applications. This fact is partly due to its ease of editing and partly due to the significant 
amount of computational complexity required by motion estimation, which results hard to 
adopt in real-time video systems. In the H.264/AVC standardization process (Richardson, 
2003) the compression performance of Intra coding was significantly improved by the adop- 
tion of spatial prediction in Intra frames, which have permitted the H.264/AVC coder to ob- 
tain a higher compression gain with respect to the previous coding standards, like JPEG2000 
(Cho et al., 2007). The pixels of the current block are predicted using the reconstructed pixels 
of neighboring blocks interpolated along different orientations, which result closely related 
to the characteristics of the image correlation (Cappellari and Mian, 2004). In the first ver- 
sion of the H.264/AVC standard (Joint Video Team, 2002), the spatial prediction is limited to 
either blocks of 4 x 4 pixels or whole macroblocks (MBs) of 16 x 16 pixels. In the FRExt ex- 
tension of the standard (Joint Video Team, 2004), blocks of 8 x 8 pixels are considered too. As 
a consequence, the computational complexity of an exhaustive rate-distortion optimization 
significantly increases because of the number of different partitioning modes and prediction 
directions. In order to overcome this problem, a wide variety of complexity reduction strate- 
gies, together with the introduction of novel hardware accelerators, have been proposed in 
literature. 

Pan et al. (2005) propose a fast Intra prediction algorithm that extracts the image features using 
Sobel edge operators and chooses the predictor according to their statistics. In a similar way, 
the approaches by Pan et al. (2004) and by Ryu and Kim (2007) extract the directional features 
of each frame and use them to estimate the most probable prediction modes. The solutions 
proposed by Xin et al. (2004) and by Jeong and Kwon (2007) evaluate the distortion produced 
by prediction in the transform domain, while (Kim et al., 2006a) suggest extracting jointly 
the features of each block from both pixels and transform coefficients. In addition, temporal 
correlation existing between adjacent frames can be used too, as it is shown by Xin and Vetro 
(2006). 

Many approaches employ early-termination decision in order to reduce the amount of compu- 
tation (Loras and Amiel, 2005). This makes the computational complexity significantly vary 
according to the processed video sequence (see the strategy proposed by Yong-dong et al. 
(2004) as an example where the relative reduction of coding time varies from 40% to 70%), 
and therefore, an a priori estimation of the resulting cost is not possible. At the same time, the 
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performance of the algorithm varies according to the coded sequence like in the case of the 
approach by Kalva and Christodoulou (2007) where a machine learning algorithm is used to 
select the best prediction mode among a reduced set of candidates. 

With respect to these methods, the design of a complexity reduction strategy that permits 
controlling the amount of required computation provides several advantages, such as 

• the possibility of adapting the algorithm to devices with different computational capa- 
bilities and power supply; 

• an accurate estimation of the autonomy of battery-powered coding devices; 

• the possibility of enabling power saving configurations that gradually reduce the com- 
putational complexity (at the cost of a worse rate-distortion optimization) according to 
the remaining battery charge. 

The solution presented in the following computes for each 4x4 prediction mode the prob- 
ability that it minimizes the cost function with respect to the other ones. According to this 
probability distribution, the algorithm elects a limited set of modes (the most probable ones) 
as possible "best-mode" candidates and computes the cost function for each of them. The prob- 
ability estimation is performed using a low-cost Belief-Propagation (BP) strategy that exploits 
the statistical dependence among adjacent blocks. In the end, the algorithm checks whether it 
is worth merging the blocks together or not. 

In the following, Section 2 will describe the Intra prediction process in the H.264/AVC FRExt 
coder and the related complexity problems. Section 3 presents how different solutions try to 
cope with the computational issue by designing appropriate low-complexity Intra prediction 
strategies. Then, Section 4 will present the proposed algorithm that estimates the best-mode 
probability for each prediction orientation from the previous coding results. The estimated 
probability distribution permits computing a reduced set of candidate modes. In second step, 
the algorithm chooses whether it is worth merging blocks together or not. Experimental re- 
sults, presented in Section 5, will show that the performance of the algorithm compares well 
with other solutions, and in addition, that the computational complexity can be controlled by 
increasing or decreasing the number of candidate modes. Final conclusions will be drawn in 
Section 6. 

2. The Intra coding mode in the H.264/AVC FRExt standard 

Like many of the previously-proposed video coders (ISO/IEC JTC1, 2001; ITU-T, 1995; ITU- 
T and ISO/IEC JTC1, 1994), the H.264/AVC standard adopts a hybrid coding scheme that 
combines traditional transform coding with a predictive coding approach (see Fig. 1). The 
adoption of low-complexity integer transform decreases the compression efficiency of the 
transformation procedure for the sake of a lower computational complexity; it is possible 
to compensate this decrement in the coding gain by predicting the input signal from pixels 
belonging to the previous frames or to the previously-coded blocks. 

The input video frames captured by the camera are partitioned into macroblocks of 16 x 16 
pixels, and each macroblock can be divided into blocks of 4 x 4 or 8 x 8 pixels which can be 
predicted according to the chosen coding mode. For Inter macroblocks (i.e. temporally pre- 
dicted macroblocks), from the previously-coded frames the coder selects a predictor block that 
approximates well the current one and identifies it using a motion vector (MV). This selection 
is performed via a motion search process, which considers all the blocks whose coordinates 
lie within a given search window and chooses the one that minimizes a given distortion met- 
ric. As for Intra macroblocks, the current block is predicted using the neighboring pixels 
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Fig. 1. Block diagram of the H.264/AVC coder. 



belonging to previously-coded blocks and interpolating their values according to a set of lin- 
ear equations. The residual signal after prediction is then transformed and quantized into a 
set of integer coefficients whose values are converted into a binary bit stream. The size of the 
adopted integer transform can be either 4 x 4 or 8 x 8 depending on the chosen macroblock 
partitioning. Since the adopted transforms are not orthonormal, the quantization unit needs 
to compensate this fact by rescaling the quantization steps for the different coefficients de- 
pending on the spatial frequencies. As a consequence, the set of quantization steps associated 
to a given distortion level is referenced using the Quantization Parameter QP, which assumes 
integer values in the range [0 51] and is exponentially proportional to the quantization step 
according to the equation 

A = K ifj 2 QP/6 (1) 

where Ku is a rescaling factor that depends on the spatial frequencies (i,j) and on the adopted 
quantization matrix. Then the block of coefficients is dequantized, inversely- transformed, 
and summed to the corresponding predictor block in order to reconstruct the coded signal. In 
the following, the chapter will be focused on the spatial Intra prediction. 
Since the earliest stages of its standardization process, the Intra coding mode of the 
H.264/AVC codec has been characterized by block-based spatial prediction. The pixels in 
the current block are predicted from the neighboring ones according to a spatial predictor 
which is chosen among a set of possible standardized candidates (see Fig. 2 as an example for 
the Intra4x4 mode). 

More precisely, each candidate predictor is computed from the neighboring pixels of the upper 
and the left macroblocks interpolated along an assigned spatial directions. The H.264/AVC 
standard defines a finite set of directions whose number can vary from 4 up to 9 accord- 
ing to the coding mode of the block. At first, two Intra coding modes were defined, named 
Intra4x4 and Intral6xl6 respectively. The first one performs spatial prediction on blocks 
of 4 x 4 pixels and has a set of 9 candidate predictors (reported in Fig. 2), while the second 
one predicts a whole macroblock of 16 x 16 pixels choosing one predictor among a set of 4 
(see Fig. 3). As for the Chroma component, only 4 modes were standardized defining an Intra 
prediction on 8 x 8 blocks. With the extension of the coding standard (H.264/AVC FRExt), a 
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novel Intra8x8 mode was introduced using 9 possible candidates on Luma blocks of 8 x 8 
pixels (Joint Video Team, 2004). Experimental results (Cappellari and Mian, 2004) have shown 
that the performance of spatial prediction coding in the H.264/AVC coder depends on the 
efficiency of the chosen directional predictor in modelling the characteristics of the signal. 
Given the Intra macroblock coding mode M (M = Intra4x4, Intra8x8, Intral6xl6), the 
default Intra coding strategy implemented in the reference H.264/ AVC coder tests for the cur- 
rent block all the possible predictor blocks associated to the set of available prediction modes 
and chooses the mode m that minimizes the Lagrangian cost function 

L(m) = D(m) + AK(m). (2) 

The value R(m) is the bit rate needed to code the current mode m in the bit stream, D(rri) is the 
distortion metric, and A is a Lagrange multiplier that weights the influence of both distortion 
and bit rate in the cost function (Sullivan and Wiegand, 1998). In finding the best prediction 
mode, the distortion D(m) is measured using the Sum of Absolute Differences (SAD) between 
the predicted block and the original one in order to limit the required computational complex- 
ity. Since the best prediction mode m for the current 4x4 block is strongly correlated with the 
modes chosen for the spatially-neighboring blocks, in the H.264/AVC standard the bit rate 
R(m) is coded after estimating the most probable prediction mode according to the modes 
of the upper and left blocks (Joint Video Team, 2004). The same rate distortion strategy is 
adopted to find the best Intra macroblock coding mode M, where the distortion metric D(M) 
is the Sum of the Squared Differences (SSD) between the original macroblock and the recon- 
structed one in place of the SAD. Depending on the adopted distortion metric (SAD or SSD), 
the value of the parameter A is linearly or quadratically proportional to the adopted quanti- 
zation step. The derivation process of the parameter A for the reference H.264/AVC coder is 
reported in a work by Wiegand and Girod (2001), where A is set to 



A 



= 0.85 2 QP/6 when using SAD and A = 0.85 2 QP/3 when using SSD. (3) 

The distortion metric in eq (2) permits choosing the coding mode that could be slightly sub- 
optimal in terms of distortion but requires a lower amount of bits. 

In the following, the chapter will give a more detailed description about each prediction mode, 
how it is chosen, and how it is coded in the bit stream. 

2.1 Spatial prediction on 4 x 4 blocks 

The baseline Intra coding mode defined within the H.264/ AVC standard is the Intra 4x4 
mode, which partitions the macroblock into 4 x 4-pixels blocks and chooses for each block 
a spatial predictor out from a set of 9 possible standardized candidates. In Figure 2 all the 
9 possible modes are reported (for a detailed description including the formulas to estimate 
the pixel values of each predictor see the standard release by Joint Video Team 2004). Note 
that each mode is associated to an identification number that is closely related to its average 
best-mode probability, i.e. the probability of being chosen as best prediction mode for the cur- 
rent block. As a consequence, vertical and horizontal modes, which are the most frequently 
adopted prediction orientations, are assigned to the numbers and 1. Since the best predic- 
tion mode m for the current 4x4 block is strongly correlated with the modes chosen for the 
spatially-neighboring blocks, in the H.264/ AVC standard the bit rate R(m) is coded after esti- 
mating the most probable prediction mode according to the modes of the upper and left blocks 
(see Joint Video Team (2004)). The variable most_probable_mode is defined as the mini- 
mum between the modes of the neighboring upper and the left blocks. Whenever estimating 
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Fig. 2. Possible predictors for Intra4x4 macroblock coding mode of H.264/AVC standard. 



the best prediction mode m for the current block, the parameter R(m) is equal to 1 or 4 since the 
coder at first signals with one bit whether the chosen mode equals most_probable_mode 
or not. In case m and most_probable_mode differs, the coder requires 3 additional bits to 
signal the correct predictors. 

2.2 Spatial prediction on 16 x 16 blocks 

The adoption of spatial predictions on 4 x 4 blocks imply that the video coder has to specify 16 
prediction modes for each Intra4x4 macroblocks with a consequent waste of coded bits. In 
addition, during the standardization process of the H.264/AVC architecture, preliminary ex- 
perimental results had shown that spatial orientation of neighboring blocks does not change 
wherever the image is highly stationary (uniform regions). In these situations, the same re- 
sults had also shown that performing spatial prediction on wider blocks proved to be more 
effective since a reduced number of prediction modes need to be specified. As a consequence, 
the H.264/AVC video coding standard was enabled with the additional Intra 1 6x1 6 coding 
mode, which predicts the whole macroblock using the pixels of the upper and the left mac- 
roblocks lying along the borders. In this case, 4 possible orientations were defined and, after 
performing the 4x4 integer transform on each residual block within the macroblock, an ad- 
ditional 4x4 Hadamard transform is applied on the DC coefficients (Joint Video Team, 2004). 
Figure 3 reports a graphic representation of the possible Intra prediction modes. Anyway, 
the extension of the H.264/AVC standard to other applications and video formats brought 
the need of defining an additional Intra prediction mode operating on the intermediate 8x8 
blocks. 



2.3 Spatial prediction on 8 x 8 blocks 

Initially conceived for video communication and video streaming applications on low band- 
width channels, the standard H.264/AVC was successively extended to the transmission and 
storage of high definition video. As a drawback, the 4x4 integer transform was no more 
suitable for wider video formats, and therefore, an additional 8x8 integer transform was in- 
cluded in the standard. In addition, experimental results showed that performing the block- 
based spatial prediction on 8 x 8 blocks proved to be effective for a wide number of mac- 
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Fig. 3. Possible predictor 



! macroblock coding mode of H.264/ AVC standard. 



roblocks. As a result, the Intra8x8 coding mode was introduced within the standard as an 
additional feature which has to be included in all the decoders implementing the highest pro- 
files (Joint Video Team, 2004). The Intra prediction on 8 x 8 blocks has 9 different prediction 
orientations similar to those defined for the 4x4 blocks. In this case, the neighboring pixels 
are initially smoothed by a low-pass Finite Impulse Response (FIR) filter and then used to 
generate the predictor blocks. The coding strategy for the prediction mode is the same for the 
mode Intra4x4 (see Fig. 4). In this case, the variable most_probable_mode can be com- 
puted from neighboring 4x4 blocks too (see the standard draft by Joint Video Team, 2004 for 
more details). 



2.4 Spatial prediction on chrominance blocks 

For input video signals in the YUV 4:2:0 format, which is the main format supported by the 
standard H.264/ AVC (however, other formats are supported), every macroblock of luminance 
pixel is associated to two 8x8 pixel blocks in the U and V components respectively. Spatial 
prediction is also performed on these blocks considering neighboring pixels from the upper 
and the left macroblock. In this case, the characteristics of the chrominance signals do not 
require a wide range of possible predictor modes to perform an effective spatial prediction, 
and therefore, only 4 modes are defined within the standard similar to those defined for the 
Intral6xl6 mode despite the fact that the indexing is changed (see Fig. 3). 
The default Intra coding algorithm tests all these possible choice and chooses the one that 
provides the best rate-distortion performance in terms of the metric of eq. (2). In the following 
we will present some solutions that permit obtaining a coding efficiency comparable to that 
of the extensive method and require a limited computational complexity. 
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Fig. 4. Possible predictors for Intra 8x8 macroblock coding mode of H.264/AVC FRExt stan- 
dard. 



3. Overview of the existing Fast Intra Prediction methods 

Most of the fast Intra coding algorithms reduce the computational complexity by identify- 
ing the spatial orientation of the current block and selecting an appropriate set of candidate 
modes without performing a complete testing of all the possible predictors (Pan et al., 2005). 
At the same time, these algorithms select the MB partitioning (Intra4x4 or Intral6xl6) 
that suits better to the current macroblock. Many methods rely on early termination solution, 
where the best predictor search is terminated before testing all the available modes whenever 
the resulting distortion is lower than a given threshold. Other solutions analyze the charac- 
teristics of the block to be coded in order to identify the most suitable prediction mode. These 
methods test the different predictors according to a hierarchical order or process the input 
block using some edge detection operators to create a set of possible candidate modes. The 
outcomes of these operations are used to infer the spatial orientations of the block to be coded 
and identify, as a consequence, the associated prediction mode. In most of the solutions pre- 
sented in literature, the number of tested modes or operations depend on the characteristics 
of the input video signal, and as a consequence, the required computational complexity varies 
without permitting an a priori estimation. In the following some of these methods are shortly 
presented. 



3.1 Fast Intra Prediction using edge detection operators 

Since the efficiency of the spatial prediction relies on identifying accurately the orientation 
of the spatial correlation, many approaches try to infer this feature from the coded video 
signal using some edge detection operators. One of the first approaches that perform fast 
Intra prediction estimation was proposed by Pan et al. (2005). When evaluating the Intra 4x4 
mode, the input 4x4 block is processed using vertical and horizontal Sobel operators, and 
the outcoming values are stored in a histogram with 9 bins associated to the different modes. 
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These values are used to estimate a set M of possible candidate modes. The mode with 
the highest probability, together with the two modes that result the closest to it and the DC 
prediction mode, is included in M. Then, the algorithm chooses the predictor that obtains 
the lowest value in the cost function. As for the Intra 1 6x1 6 mode, only the most probable 
mode and the DC mode are considered reducing the number of tested modes from 4 to 2. 
When coding the 8x8 chrominance blocks, the most probable modes are estimated for both 
the U and V components from the orientation histograms and included in M., together with 
the DC mode. 

A similar approach is adopted in another work by the same authors (Pan et al., 2004), where 
an average edge directional field is computed for each 4x4 block in order to identify the 
dominant spatial orientation. The quadratic values of the Sobel operators are averaged within 
the current block in order to estimate the dominant spatial direction and the related coherence 
value. 

A different approach is proposed by Yong-dong et al. (2004), which generates a subsampled 
version of the current 4x4 block and computes vertical and horizontal edge detection op- 
erators. According to the absolute values and the signs of these, different sets of candidate 
predictors (with different numbers of modes included) are generated and tested. In this way 
it is possible to save about 60 % of the computational complexity on average, but the actual 
saving depends on the chosen quantization parameter QP. 

Other approaches approximate the distortion measure using alternative metrics, which either 
requires a lower complexity or proves to be more effective in identifying the orientation of the 
current block. The approach proposed by Kim et al. (2006b) evaluates a group of SAD metrics 
on a reduced set of pixels which are located close to the borders of each block. A possible 
alternative is presented by Jeong and Kwon (2007) where the orientation of the block is found 
computing a distortion metric on the relevant transform coefficients of the current block. 

3.2 Fast Intra Prediction using hierarchical search 

Another set of solution proposed in literature rely on the possibility that a tested prediction 
mode is very likely to be the best one whenever the associated distortion value is lower than 
a discriminating threshold. As a consequence, these solutions aim at finding the mode order 
that places the most probable best candidates first. 

The approach by Lu and Yin (2005) tests the available prediction modes and coding options 
according to a predefined order. More precisely, the Intral6xl6 mode is considered at first, 
and the coding strategy tests the DC mode checking whether the associated cost function has 
a lower value with respect to fixed discriminating threshold. In case the cost is higher, vertical 
and horizontal modes are tested as well; otherwise, the Intral6xl6 coding process is fin- 
ished {early termination). The Intra4x4is then tested considering DC, vertical, and horizontal 
modes at the beginning. In case the cost function for the Intral 6x1 6 mode is lower than a 
given threshold no additional Intra4x4 prediction modes are considered. 1 Otherwise, the 
algorithm tests the remaining prediction orientations that are closer to the best mode between 
the vertical and the horizontal ones. The presence of early termination decision does not allow 
an accurate a priori estimation of the required computational cost. 

In a similar way, the solution designed by Loras and Amiel (2005) tests the vertical and the 
horizontal directions first, and according to whether the vertical or the horizontal orientation 
is better, it chooses the following set of modes to test. The same policy is applied to the new 



1 Early termination for the DC mode is evaluated also in this case. 



Spatial prediction in the H.264/AVC FRExt coder and its optimization 249 



candidate modes following a tree-ordered refinement policy of the Intra prediction for the 
current block. 

Another hierarchical solution was proposed by Kalva and Christodoulou (2007), where the 
modes are tested following an adaptive tree structure that is modified using a machine learn- 
ing algorithm. 

3.3 Fast Intra Prediction using parametric models 

Among the strategies that reduce the required computational complexity, a separate mention 
has to be done for those strategies that aim at achieving a lower computational cost the cod- 
ing performance of the rate-distortion optimization algorithm proposed within the standard 
(Wiegand and Girod, 2001). The rate and the distortion of the final coded block are estimated 
using some parametric models. This class of algorithms compute some low-complexity met- 
rics that characterize the features of the original signal, and use them to estimate the final 
coded bit rate and the associated distortion, whose calculation requires a significant amount 
of operations. 

The approach proposed by Kim et al. (2006a) estimates the possible results of the rate- 
distortion optimization algorithm from the SAD metric computed on the pixel blocks and 
on the blocks obtained after a Hadamard transform (in this case the SAD is called SATD). The 
SAD and the SATD values permit identifying the prediction mode that is the most likely to be 
the best one. In a similar way, the strategy by Kim et al. (2003) infers a statistical model for the 
current block from the SATD values. 

Unfortunately, many of these solutions adopt early termination strategies that make the re- 
quired computational complexity vary. In the following we will present an optimization ap- 
proach that permits controlling the amount of calculation with deterministic accuracy. In this 
way, it is possible to configure the algorithm in a flexible way according to the desired com- 
putational complexity. 

4. A low-complexity Belief Propagation based Intra prediction strategy 

The approach proposed by Milani (2008) reduces the set of tested candidate modes according 
to a probability estimation strategy, which is based on a Belief Propagation algorithm. This 
solution can be divided into three parts. At first, the algorithm estimates the most probable 
orientations for the current block. The estimated probabilities are used to generate a set of 
candidate predictors, and the best prediction mode is found by coding the current MB using 
the Intra4x4 mode. In the following, the 4x4 blocks are fused into either 8x8 blocks or a 
whole 16 x 16-pixels macroblock according to their orientations. The following sections will 
present the three phases in detail. 

4.1 Probability estimation for the best candidate modes 
4.1 .1 Estimation of orientations for 4 x 4 blocks 

Assuming that the Mq x 1 array p(x,y) = [p m (x,y)] (m = 0, . . . , Mq — 1) groups the probabil- 
ities p m (x,y) that the mode m is the best mode for the block at coordinates (x,y) (with Mq the 
total number of candidate modes), it is possible to write the elements of p(x,y) as follows 

p m {x,y) = p T (x,y - 1) Q m {x,y) p(x - l,y), (4) 

where Q m (x,y) = [qf:(x,y)\ is an Mq x Mq matrix. The value c\fAx,y) represents the condi- 
tional probability that mode m is the best mode for the current block at (x,y) given that i and 
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j are the best modes for blocks at coordinates (x,y — 1) and (x — \,y) respectively. However, 
it may happen that only a smaller set M of M candidate modes (M < Mq) are available for 
the block at (x,y), and therefore, the probabilities p m >(x,y) are for m' £ M. This is the 
case of blocks placed at positions where some reference pixels are not available because of 
the frame boundaries or the block coding order (e.g. upper-right pixels can not be used since 
the corresponding neighboring block has not been coded yet). The same candidate modes 
reduction is found for all the blocks whenever the H.264/AVC coder adopts a fast intra pre- 
diction algorithm that tests only a selected set of candidates to constrain the computational 
complexity. This candidate modes reduction affects the best-mode probability array, which 
can be replaced with the relation 

p(x,y) = P M (x - \,y) p(x,y) (5) 

where Pm (x, y) is a singular projection matrix that sets to some elements of p(x, y) according 
to which candidate modes are available. 

As a consequence, the best-mode statistics for the current block at position (x,y) can be esti- 
mated propagating the best-mode probability of previous blocks via the equation 



p m {x,y) = V T (x,y-l)Pl [ (x,y-l)Q m {x,y)P u (x-l,y) V {x-\,y) 
= p T (x,y-l)Q m (x,y)p(x-l,y) 



(6) 



(which is a modified version of eq. (4)), and projecting the array p(x,y) onto the subspace 
of allowed modes using equation (5). The resulting array p(x,y) differs from the original 
estimate p(x, y) of eq. (4) because of the approximation introduced by the projection and leads 
to a different set M ^ M of candidate modes. As a possible drawback, the chosen predictor 
could not match accurately the orientation of the local correlation either because the optimal 
mode is not included in the set M or because all the required neighboring pixels are not 
available and the most appropriate predictor can not be adopted. The finally chosen mode in 
could result sub-optimal for the current block and is going to affect the accuracy of probability 
estimation for the following adjacent blocks. It is possible to mitigate this effect by adopting a 
Belief-Propagation (BP) strategy that refines the statistics for each block. 

4.1.2 The Belief-Propagation procedure for spatial orientations of 4 x 4 blocks 

Before coding the block at the coordinates (x,y), the mode estimation routine propagates 
through a BP procedure the information about the best modes for the upper and left blocks 
found during the coding operations (see Figure 5 for a graphic example). These modes are 
denoted here with m[x,y — 1) and ih(x — l,y) respectively. According to this, the coding rou- 
tine estimates a probability mass function (pmf) p(x,y) for the current block via equation (6), 
where 



p m {x i,yj- | 1 m = ih ^ x _ lfy) 



(7) 

v (xv-D-l ° m ^ ih ^v~ 1 ) 

p m {x,y lj-| 1 m = m ( X/y _ 1 y 

According to the values of p(x,y), all the possible prediction modes are sorted in decreasing 
probability order, and the most probable ones are included in the set M according to the 
criteria that will be described in Section 4.2. 
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Fig. 5. Probability propagation according to the implemented Belief Propagation approach. 
Some message passing propagates hard information (solid arrows) regarding the chosen pre- 
diction modes, while others communicates likelihoods associated to the prediction mode of 
4x4 blocks (dashed arrows). 



After finding the mode that minimizes the cost function among the candidates in M, the BP 
approach propagates this result to the previously coded blocks in order to refine the accu- 
racy of the estimated mode probability (i.e. p(x,y — 1) and p(x — l,y)). The array p(x,y), 
whose elements are reported in eq. (7), is replaced by a "soft" best mode estimation p(x,y) (a 
likelihood) computed using a reversed version of equation (4) 



P(x,y) 



p T (x,y-l)Q^(x,y)p(x + l,y). 



(8) 



The new arrays p(x,y) affect the estimated mode probability distribution for the following 
blocks and improve the compression performance of the fast Intra coding algorithm. As an 
example, the elements for the probability array p{x,y + 1) of block 9 in Fig. 5 are obtained via 
eq. (6) replacing the array p[x,y — 1) with p{x,y — 1). 

Experimental results have proved that the refinement step performed using equation (8) does 
not change the arrays p{x,y) in such a way that the order of candidate modes is altered. 
However, the likelihood estimate for the prediction mode m[x,y) proves to be significant in 
the computation of the number of candidate modes as it will be explained in Subsection 4.2.1. 
Moreover, the best prediction modes found for Intra4x4 coding are used to characterize 
the best-mode probability of prediction modes for bigger blocks in case the rate-distortion 
algorithm has chosen to merge the 4x4 blocks together, as it will be described in Section 4.4. 
In the estimation routine, the arrays p[x,y) and p[x,y) are approximated using a finite set 
V of 100 pmfs, which has been obtained from an extensive set of training sequences via an 
LBG iterative classification (Gersho and Gray, 1991). In this procedure the distortion metric to 
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minimize is the Jensen-Shannon divergence between p(x,y) and pG? 

JSD(p(x,y)\\p) = ±D(p(x,y)\\p) + ±D(fi\\p(x,y)) (9) 

where D (p(x,y) ||p) is the Kullback-Leibler divergence 

D{p{x,y)\\p)= Ep.(x,y)logf£4^)- < 10 ) 

m=0 \ Vm J 

The conditional probability matrix Q m (x,y) is a linear combination of arrays pG?, and the 
probability array p(x,y) at position (x,y) is updated after each iteration of the BP procedure 
using a Finite State Machine (FSM), where each state is related to an element of V . In this way, 
it is possible to obtain an adaptive estimation of the probability for each prediction mode with 
limited computational complexity and memory area. 

4.1.3 Estimation of probable spatial orientations for 8 x 8 blocks 

After the optimization algorithm has chosen to merge together 4x4 blocks into 8x8 blocks, 
the coder estimates an Intra8x8 best-mode probability distribution p 8x8 according to the 
previously found Intra4x4 modes. The adopted approach estimates three different mode 
probability distribution p 8x8 ' f , i = v,h,d, which are dependent on the best Intra prediction 
modes of vertical, horizontal and diagonal couples of 4 x 4 blocks respectively (see Figure 6). 
Using the same notation of equation (4), it is possible to write p 8x8 ' z = [p m x ,l ],i = v,h,d and 
m = 0, . . . , 8, as 

p^ = p T (x,y)F 8 m ^p(x,y + l) 

p^ h = p T ( X ,y)F s m ^ h p(x + l,y) (n) 



n Sx8,d 



p T (x,y)F^ d p(x + l,y + l) 



where p(x,y) represents the chosen prediction mode (as defined in equation (7)) and F m x ,l , 
i = v, h, d, is the conditional probability matrix of 8 x 8 Intra prediction mode m given the 
vertical, horizontal, and diagonal couples of 4 x 4 modes. In this way, Intra8x8 best-mode 
probability estimation relies on the results of Intra4x4 coding which has already been per- 
formed on the current macroblock. 

4.2 Estimation of the set of candidates 

4.2.1 Computation of the most probable prediction modes 

After estimating the probability array p{x,y), the coding routine has to identify those modes 
that are more likely to be the best prediction mode for the current 4x4 block. The num- 
ber of candidate modes M is usually set to the average value M, but can vary according to 
the characteristics of the probability distribution identified by p(x,y). In fact, experimental 
data show that the entropies of distributions p(x,y) vary, and therefore, the mode probability 
distributions with a lower entropy only needs a reduced number of candidates. 
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Named fi = [fi m ] the average mode probability array, M is chosen in such a way that 

M-l M-l 

E Smipfry)) < £ Sm(«(x /y )) 

ra=0 m=0 /i 2 s ) 

M M-l V ; 

E s «(p(^y)) > E M*(x, y )) 

ra=0 m=0 

where S m (•) : [0, l] 9 — >■ [0, 1] is an ordering function that returns the m-th value of the input 
array in decreasing order. The value M reports the average number of modes to be tested 
for each block and permits controlling the computational complexity In this way it is pos- 
sible to provide the same probability of finding the best prediction mode with a limited sets 
of candidates to all the 4x4 blocks of the image. This equalization permits saving some 
computational complexity without affecting the coding performance of the algorithm. 
As it was mentioned in Subsection 4.1.2, the refinement provided by either p(x,y + 1) or 
p(x + 1,1/) permits a better estimate of the probabilities related to the candidate modes of the 
current block. This improvement does not lead to a change in the order of modes but could 
modify the number of candidates that is considered for the current 4x4 block since it affects 
equation (12). Experimental results have shown that the refinement brought by the Belief- 
Propagation strategy leads to a reduction of the coding time with respect to the case when 
forward message passing is allowed only (see the solid-line arrows in Fig. 5). The same ap- 
proach is adopted to estimate the set of candidate modes for Int ra8x8 as it will be described 
in Subsection 4.3. 

4.2.2 Further reduction of the possible candidates (DD algorithm) 

According to the probability values of p(x,y), the M most probable modes are included in 
the set M of candidates. Whenever the entropy associated with p(x,y) is high, it is possible 
that the set M includes modes with orthogonal spatial orientations. This fact is mainly due to 
the transient period in the probability estimation process, which may require several iteration 
before converging to an accurate estimate of mode statistics. Therefore, a further reduction 
of the candidate modes can be obtained by estimating whether horizontal or vertical modes 
are dominant in the distribution p(x,y) and eliminating the dominated modes (Dominated 
Deletion - DD). The number of orientations in the set M which are close to the vertical one is 
compared with the number of candidate modes which have a spatial orientation close to the 
horizontal one. In case one of them prevails, the modes of the other type are deleted from the 
set M. 

This additional improvement proves to be quite effective whenever the image orientation 
statistics has changed, and the array p(x,y) estimated by the algorithm does not provide a 
sufficiently-accurate approximation of the real pmf yet. Therefore, the estimated set M could 
include some candidate modes which are orthogonal to the other ones since they could be 
probable candidates in the neighboring region. The DD elimination algorithm prevents this 
transient phase from reducing the effectiveness of the algorithm and speeds up the best intra- 
prediction estimation process. 

However, this elimination procedure has to be constrained in order to avoid an excessive 
reduction of the candidate sets whenever the statistics of prediction orientations is not clearly 
biased on either vertical or horizontal directions. In order to avoid the deletion of probable 
candidate modes, the DD algorithm is performed only for modes greater than 4 whenever 
the number of dominated modes is greater than a certain threshold value T. In the setting 
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Fig. 6. Merging operation for 4x4 blocks. 
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operation of the fast intra algorithm, the parameter T can be varied in order to increase or 
decrease the cardinality of M according to the desired computational complexity. 
Experimental results reported in Section 5 will underline its contribution in the overall perfor- 
mance of the fast intra prediction procedure. 

4.3 Computation of the candidates for 8 x 8 blocks 

In case the coding mode Intra8x8 is enabled, the fast intra prediction algorithm has to es- 
timate the most appropriate prediction modes for the current 8x8 block from the results of 
Intra4x4 mode. The transcoding algorithm reported by Bialkowski et al. (2004) chooses the 
most frequently used prediction direction, but in case the estimated 4x4 orientations prove 
to be nonuniform within the same 8x8 block, a better performance can be obtained testing a 
set M$ X 8 of different candidates. A procedure similar to that of Subsection 4.2.1 is adopted in 



order to estimate the sets of candidate .MsxS f° r the current 8x8 block from p 



,8x8,z' 



v, h, d. 



Each mode probability distribution p 8x8 ' z infers a different set M.$ x $ f i, z = v,h,d, of candi- 
date modes which is obtained in the same way of the set of M possible candidate modes for 
Intra4x4 blocks. In case the set .Ms x 8 obtained from the intersection of the sets 



A4sx8 = A^8x8,i; H M 8x8/h H M 8x8/d 



(13) 



is not empty, the coding algorithm merges the 4x4 blocks into a 8 x 8 block and tests the 
predictors included in the set .Ms x 8 looking for the one that minimizes the cost function. 
As for the Intral6xl6 coding, all the four possible predictions are tested since the estimation 
of the best mode probability for the 16 x 16 block from the best Int ra4x4 modes is not trivial. 
The same choice is adopted to spatially predict the chrominance components U and V. 

4.4 Estimation of best macroblock partitioning for Intra prediction 

After finding the best mode for each 4x4 block in the current MB, the coding routine tests 
whether it is better to use bigger blocks. In a first step the algorithm checks whether it is 
possible to merge together the 4x4 blocks into blocks of 8 x 8 pixels. In case the orientations 
of each 4x4 block are the same or close, the merging of separate blocks results convenient 
with respect to the Intra4x4 block partitioning since a reduced number of predictors needs 
to be coded in the transmitted bit stream. 

In order to detect these configurations, the encoder estimates the orientation differences 
d(fh(x,y),m(x + i,y + /)) {i,j = 0,1) between vertical, horizontal and diagonal couples of 
4x4 blocks (see Figure 6) within the current 8x8 block. The metric d(m(x,y),m(x! ' ,y')) is 
computed as follows 



d{ih{x,y),m(x' ,y')) = \Zm(x,y) — Am(x' ,y')\ 



(14) 
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Fig. 7. Block diagram for the general Fast Intra algorithm. 



where /-m denotes the angle associated to the spatial orientation of mode m. If the average 
difference 

- _ d(m(x,y),m(x + l,y)) d(m(x,y),m(x,y + 1)) d(fh(x,y),m(x + \,y + 1)) 

a ~ 3 + " 3 " + 3 y } 

is lower than 40°, the 4 x 4 blocks at (x,y), (x + 1,1/), (x,y + 1), and (x + l,y + 1) could be 
merged into one block of 8 x 8 pixels. In case the condition on d is verified for all the 8x8, 
the Intra8x8 coding mode is enabled. Moreover, the encoding routine tests whether it is 
worth merging the 8x8 blocks into one common 16 x 16 prediction block considering the 
Intra4x4 modes for the blocks at the border of 8 x 8 blocks. In case the average absolute 
difference between the orientations of 4 x 4 blocks lying at the borders of 8 x 8 blocks is lower 
than 40°, the Intral6xl6 prediction mode is chosen for the current macroblock. In this way, 
the wider block partitioning modes are tested only in case the orientations for the 4x4 blocks 
are approximately uniform, otherwise either 4 x 4 or 8 x 8 partitioning is preferred. 
The whole fast Intra coding procedure is depicted in the block diagram of Fig. 7 and can be 
summarized by the following pseudo-code: 



Test Intra4x4 coding mode 

for each 4x4 block in the current macroblock do 

compute p(x,y) 

compute M and create the set M 

test all the modes in M an 
end for 
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7: check if it is worth merging the 4x4 blocks into bigger blocks as described in the current 
Section 

8: if Intra8x8 is to be enabled then 

9: for each 8x8 block in the current macroblock do 
10: compute p 8x8 ' z , { = v,h,d 

11: compute M$ X 8 anc * find the best mode 

12: end for 
13: end if 

14: if Intral 6x1 6 mode is to be enabled then 
15: find the best prediction mode 
16: end if 
17: choose the MB Intra coding mode that minimize the total cost function. 

Experimental results will show that this choice leads to good performance with respect to 
other proposed solutions. 

5. Experimental results 

In order to test the efficiency of the presented algorithm, different sequences were coded 
with different quantization parameter values and enabling different Intra coding modes. The 
proposed Intra coding strategy was implemented into the JM10.1 software. In the tests the 
adopted parameter setting is the same of the paper by Pan et al. (2005), coding different 
sequences with only Intra frames and QP = 28, 32, 36, 40. At first the performance of 
Intra4x4 and Intral 6x1 6 modes only was evaluated, comparing the computational com- 
plexity, the PSNR value, and the coded bit rate of the presented solution with those provided 
by the full-complexity rate-distortion optimization algorithm implemented in the reference 
software. Experimental data show that the PSNR vs. rate curves of the two methods are quite 
close (see Figure 8). It is possible to notice that coding performance in terms of rate-distortion 
optimization is related to the target number M of candidate modes, which can vary according 
to the available computational resources or the remaining power supply. 
Table 1 reports the PSNR loss, together with the rate increment and the complexity reduction, 
for the proposed approach with respect to the reference software (with Intra8x8 mode dis- 
abled). The presented algorithm is able to reduce the coding time of approximately 63% with 
respect to the JM exhaustive approach with an average rate increment lower than 5% and a 
PSNR loss of 0.16 dB (M = 6 and T = 2). The DD algorithm described in Subsection 4.2.2 
makes possible to improve the relative coding time reduction of an additional 13% (compare 
results for M = 6 and T = 2 with results for M = 6 without DD). 

The reported data also show that the rate-distortion performance is slightly better than that of 
the approaches by Pan et al. (2005) and by Yong-dong et al. (2004). The bottom part of Table 1 
reports the results for the algorithm proposed by Pan et al. (2005). Equalizing the rate incre- 
ment, the performance of the proposed algorithm with M = 7 and T = 2 permits reducing 
the PSNR loss of 0.04 dB and improving the coding time saving of approximately 2%. Despite 
this slight improvement, the real advantage of the proposed approach relies on the possibility 
of forecasting the computational complexity required by coding operations. Table 2 reports 
the range of variation for the saved coding time of different fast Intra coding algorithms and 
different configurations. It is possible to notice that the computational complexity does not 
significantly vary according to the input sequence, since the maximum deviation of time sav- 
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(M,T) 


Sequence 


A Bits (%) 


APSNR 


A Time (%) 




container (qcif) 


10.42 


-0.18 


-72.24 




news (qcif) 


9.42 


-0.23 


-72.44 


(5,2) 


coastguard (qcif) 


8.04 


-0.18 


-71.55 




bus (cif) 


6.22 


-0.21 


-70.50 




tempete (cif) 


4.55 


-0.30 


-69.42 


average 


7.73 


-0.22 


-71.23 




container (qcif) 


5.35 


-0.14 


-62.56 




news (qcif) 


5.79 


-0.16 


-62.46 


(6,2) 


coastguard (qcif) 


3.55 


-0.14 


-62.64 




bus (cif) 


2.15 


-0.16 


-63.64 




tempete (cif) 


4.95 


-0.22 


-63.00 


average 


4.36 


-0.16 


-62.86 




container (qcif) 


5.32 


-0.14 


-59.95 




news (qcif) 


5.68 


-0.15 


-60.43 


(6,3) 


coastguard (qcif) 


3.67 


-0.14 


-60.48 




bus (cif) 


2.15 


-0.16 


-63.64 




tempete (cif) 


4.95 


-0.22 


-63.00 


average 


4.35 


-0.16 


-61.50 




container (qcif) 


3.77 


-0.13 


-58.98 




news (qcif) 


4.55 


-0.16 


-57.85 


(7,2) 


coastguard (qcif) 


2.14 


-0.13 


-59.21 




bus (cif) 


2.11 


-0.14 


-58.19 




tempete (cif) 


4.61 


-0.19 


-57.88 


average 


3.44 


-0.15 


-58.42 




container (qcif) 


3.60 


-0.12 


-55.18 




news (qcif) 


4.50 


-0.14 


-54.14 


(7,3) 


coastguard (qcif) 


2.19 


-0.12 


-55.25 




bus (cif) 


2.13 


-0.13 


-54.83 




tempete (cif) 


4.57 


-0.18 


-54.36 


average 


3.40 


-0.14 


-54.75 




container (qcif) 


5.64 


-0.08 


-50.43 




news (qcif) 


4.96 


-0.09 


-47.79 


6 without DD 


coastguard (qcif) 


3.64 


-0.09 


-51.10 




bus (cif) 


2.73 


-0.10 


-49.41 




tempete (cif) 


5.05 


-0.13 


-50.07 


average 


4.40 


-0.10 


-49.76 




container (qcif) 


3.69 


-0.23 


-56.36 




news (qcif) 


3.90 


-0.29 


-55.34 


Pan et at. 


coastguard (qcif) 


2.36 


-0.11 


-55.03 




bus (cif) 


3.85 


-0.10 


-58.12 




tempete (cif) 


3.51 


-0.23 


-57.70 


average 


3.46 


-0.19 


-56.51 



Table 1. Experimental results with Intra8x8 disabled and only Intra frames. 
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Fig. 8. PSNR vs. rate for only Intra coded sequences with different target number M of candi- 
dates (Intra4x4 mode only). 



ing from its average with M = 7 and T = 2 is 0.79% while it is equal to 5.81% for the algorithm 
of Pan et at. and 30.38% for the solution proposed by Yong-dong et al. (2004). 
Moreover, it is possible to tune the parameters of the fast estimation algorithm in order to 
vary the required computational complexity and the rate-distortion performance. The results 
reported in Table 1 show that reducing the parameter M by 1 permits a decrement of the 
computational complexity between 4.44% and 8.37%. 

In addition, Table 3 reports some experimental results obtained enabling the Intra 8x8 cod- 
ing mode too. In this case the average performance does not significantly change, but the 
complexity reduction results slightly more variable because of the increased number of cod- 
ing modes. Note also that the computational saving increases since the rate-distortion opti- 
mization process becomes more complex as the mode Intra8x8 is added, and therefore, the 
adoption of fast method for Intra prediction proves to be an effective strategy in the coding 
process. 
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Algorithm 


E [ATime (%)] 


range for ATime (%) 


BP M = 5 T = 2 




-71.23 


[-72.44, -69.42] 


BP M = 6 T = 3 




-61.50 


[-63.64, -59.95] 


BP M = 7 T = 2 




-58.42 


[-59.21, -57.85] 


Pan et al. Pan et al. (2005) 




-59.57 


[-65.38, -55.03] 


Yong-dong et al. Yong-dong et al. 


(2004) 


-60.38 


[-68.70, -40.30] 



Table 2. Experimental results of different algorithms for only Intra mode. 



(M,T) 


Sequence 


A Bits (%) 


APSNR 


A Time (%) 




container (qcif) 


5.93 


-0.14 


-66.97 




news (qcif) 


6.40 


-0.19 


-64.63 


(6,3) 


coastguard (qcif) 


5.09 


-0.18 


-66.51 




bus (cif) 


3.42 


-0.20 


-63.69 




tempete (cif) 


5.66 


-0.22 


-61.78 


average 


5.30 


-0.18 


-64.72 




container (qcif) 


4.27 


-0.13 


-62.76 




news (qcif) 


5.34 


-0.17 


-59.49 


(7,2) 


coastguard (qcif) 


3.40 


-0.16 


-62.37 




bus (cif) 


3.12 


-0.17 


-59.73 




tempete (cif) 


4.99 


-0.20 


-58.02 


average 


4.23 


-0.17 


-60.47 



Table 3. Experimental results with Intra8x8 enabled and only Intra frames. 



Final tests were devoted to evaluate the impact of the BP-based fast Intra prediction on the 
complexity of the overall coding process. To this purpose, the performance of the proposed 
fast intra algorithm was evaluated enabling Inter coding modes. In this case, the Intra cod- 
ing is applied while coding Intra frames in order to find which is the best coding mode 
for the current macroblock (see the rate-distortion optimization routine of H.264/AVC by 
Joint Video Team (2004)). Table 4 reports the coding results for GOP of 100 frames with struc- 
ture IP. . .P. It is possible to notice that the proposed method improves the results of the al- 
gorithm by Pan et al. both in terms of rate-distortion performance (we obtained lower rate 
increment and quality decrement for M = 6) and of complexity reduction (the proposed ap- 
proach permits an average 25.88% reduction in the coding time with respect to the 23.57% 
reduction of the algorithm by Pan et al. (2005)). For the sake of completeness, Table 5 reports 
the results for GOP IP. . .P and Intra8x8 mode enabled. In this case the reduction of com- 
putational time saving approximately varies from 6.25% to 7.4% with respect to the approach 
with Intra8x8 disabled (see Table 4) since the proposed algorithm significantly mitigates the 
computational load of the rate-distortion optimization routine (compare the results for M = 6 
and T = 3). 



6. Conclusions 

The chapter has described the block-based spatial prediction strategy adopted within the 
H.2634/ AVC FRExt standard and stated the problem of enabling a low-complexity Intra cod- 
ing on mobile devices. An overview of different techniques has been presented underlying 
the characteristics of each solution and the required complexity. Since most of the proposed 
solutions permit obtaining a varying computational savings which depends on the charac- 
teristics of the coded signal, the focus is centered on finding a fast Intra coding strategy that 
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(M,T) 


Sequence 


A Bits (%) 


APSNR 


A Time (%) 




container (qcif) 


1.19 


-0.04 


-25.00 




news (qcif) 


1.00 


-0.05 


-25.70 


(6,3) 


coastguard (qcif) 


0.03 


-0.01 


-27.00 




bus (cif) 


0.23 


-0.01 


-26.88 




tempete (cif) 


0.44 


-0.01 


-23.82 


average 


0.58 


-0.02 


-25.68 




container (qcif) 


0.74 


-0.02 


-18.67 




news (qcif) 


0.79 


-0.06 


-19.14 


(7,2) 


coastguard (qcif) 


0.03 


-0.01 


-20.26 




bus (cif) 


0.11 


-0.00 


-23.16 




tempete (cif) 


0.38 


-0.01 


-22.20 


average 


0.41 


-0.02 


-20.69 




container (qcif) 


1.80 


-0.08 


-20.78 




news (qcif) 


1.23 


-0.07 


-23.11 


Pan et al. 


coastguard (qcif) 


0.50 


-0.02 


-21.20 




bus (cif) 


0.32 


-0.01 


-26.05 




tempete (cif) 


0.81 


-0.03 


-26.72 


average 


0.93 


-0.04 


-23.57 



Table 4. Experimental results with Intra8x8 disabled GOP IP. . .P. 



permits controlling the amount of tested modes (and, as a consequence, the required amount 
of calculation). The chapter presents a coding strategy that identifies a set of probable can- 
didates calculating their best-mode probability. The probability estimates are obtained via 
Belief Propagation strategy that relies on the statistical dependence existing between spatially 
neighboring blocks. At the same time, the presented algorithm tries to identify the macroblock 
partitioning mode that better suits the current macroblock according to the coding results of 
the Intra 4x4 mode. Experimental results compare different algorithms and show that the 
Belief Propagation based strategy obtains a significant saving in terms of coding time (approx- 
imately 62%) with a negligible decrement of the PSNR value and a small average increment 
(less than 5.14%) in the bit rate. Moreover, the presented strategy permits an accurate con- 
trol on the encoding complexity, which does not significantly vary depending on the input 



(M,T) 


Sequence 


A Bits (%) 


APSNR 


A Time (%) 




container (qcif) 


1.25 


-0.04 


-31.59 




news (qcif) 


1.05 


-0.07 


-30.78 


(6,3) 


coastguard (qcif) 


0.22 


-0.01 


-32.10 




bus (cif) 


0.25 


-0.01 


-33.70 




container (qcif) 


1.25 


-0.04 


-31.59 


average 


0.80 


-0.03 


-31.95 




container (qcif) 


0.72 


-0.04 


-27.64 




news (qcif) 


0.97 


-0.07 


-27.32 


(7,2) 


coastguard (qcif) 


0.15 


-0.01 


-27.95 




bus (cif) 


0.17 


-0.01 


-30.02 




container (qcif) 


0.72 


-0.04 


-27.64 


average 


0.55 


-0.04 


-28.11 



Table 5. Experimental results with Intra8x8 enabled GOP IP. . .P. 
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video sequence and can be tuned according to the power supply level and to the available 
computational resources. 
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Detection of Signals in Nonstationary Noise via 
Kalman Filter-Based Stationarization Approach 
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1. Introduction 

Needless to say, the signal detection is one of the most important problems in the signal pro- 
cessing area for a long time, and a great deal of investigations has been done up to the present 
time. Most of the conventional approaches are based on the (binary) hypothesis-testing, and 
treat the corrupting (additive) noise as a stationary random process because stationary process 
is rather easy to handle and moreover its (invariant) statistical parameters can be readily calcu- 
lated under the ergodic hypothesis. However, it will be no doubt that the actual random noise 
such as environmental noise is considered to be nonstationary because its statistical properties 
are not always unchanged but vary according to underlying physical circumstances. 
Thus the problem of detecting signals in nonstationary random noise is the more important. 
For such problem, several interesting methods have been proposed. For example, Hay kin 
(1996) and Haykin & Bhattacharya (1997) treat this problem and proposed a method named 
the modular learning strategy which incorporates such three fundamental blocks as time- 
frequency analysis, feature extraction and pattern classification. Also, Haykin & Thomson 
(1998) proposed an adaptive detector based on learning for the detection of the target signal 
buried in nonstationary background noises. 

Philosophically different from their method, the authors have proposed an approach to the 
signal detection in nonstationary random noise, a new method of stationarization of the ob- 
servation noise. The key of the approach is to convert the nonstationary random noise to a 
stationary one, and this procedure was named as stationarization of the observation data. 
In Ijima, Okui & Ohsumi (2005) and Ijima, Ohsumi & Okui (2006), the signal detection is per- 
formed by testing the stationarized observation data whether there is some non-stationarized 
portion or not, based on the KM^O-Langevin equation (which is the AR model with time- 
varying coefficients). If there exists such a portion in the data, the existence of a signal is 
decided. Related to the signal detection, the stationarization approach is also used in Ijima, 
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Ohsumi & Yamaguchi (2006) to estimate the time-delay of signals in nonstationary random 
noise, incorporated with the Wigner distribution-based maximum likelihood estimation. 
In this paper the signal detection problem is investigated using the stationarization approach 
to nonstationary data. The model of the corrupting noise is given by an ARMA(p, q) model 
with unknown time-varying coefficients. These coefficient parameters are estimated from the 
(original) observation data by the Kalman filter. 



2. Problem Statement 

Let {y(k)} be the (scalar) observation data taken at sampling time instant fy (k = 1,2, • • • ), 
and assume that it can be expressed as 

y(k)=s(k) + n(k) (k = l,2,...), (1) 

where s(-) is a signal to be detected, whose form is surely known, and is assumed to exist 
in a brief interval if it exists; and n(-) is the nonstationary random noise. In consequence, 
the observation data {y(k) } becomes nonstationary, but its trend time series is assumed to be 
removed by the process 

y(k) = A d Y(k), (2) 

where Y(k) is the original data received by the receiver; AY(k) = Y{k) — Y{k — 1); and d 
indicates the order. 

In this paper the random noise n(k) is assumed to be given as the output of ARMA(p, q) 
model with time-varying coefficient parameters: 

n(k) + £ *i(k)n(k - = E P/(*M* - /') + w (*)/ ( 3 ) 

i=i y=i 

where w(-) is the white Gaussian noise with zero-mean and variance parameter a 1 ; {#;(•)} 
and {j8y(-)} are slowly and smoothly varying parameters to be specified. 
Then our purpose is to propose a method of detecting the signal s(k) from the noisy observa- 
tion data {y(k)}. 

The procedure taken in this paper is as follows: 

(i) First, based on the noise model (3), coefficient functions {#;(•)} and {/3y(-)} are estimated 
using Kalman filter from the observation data {y(k)}. 

(ii) Using the estimates {&;(•)} an d {/>/(•)} obtained in (i), the observation data y(k) is modi- 
fied to become stationary. This procedure is called the stationarization of observation data. 
(iii) Using the stationarized observation data y(k), the signal detection is based on the model 

m = §(k) + w(k), (4) 

where s(k) is the modified signal. Equation (4) is familiar in the conventional signal detection 
problem where the noise is stationary. 
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3. Stationarization of Observation Data 

Recalling the assumption that the duration of the signal s(k) is short, neglect the signal in the 
observation data and consider the signal-free case, i.e., y(k) = n(k), then the observation data 
y{k) is expressed by (1) and (3) as follows: 



y{k) = - £ « f -(k)y(* - + E 0/(*M* " ;) + w «- 

i=\ ;=1 



(5) 



In order to estimate the time-varying parameters {&i (k) } and {/3y (fc) } in (5), suppose that they 
change from step k — 1 to k under random effects {e. (k)}. Define vectors 



x(k) 



-x p (k) 



, p(fc) 



- Cl (fc) 

ep+q{ k ) 



(6) 



Then, {a z -(/c)} and {ftj(k)} are subject to the dynamics, 

x(k + l) = x(k) + v(k), 

where {e. (k)} are assumed to be Gaussian with zero-means and variances t\, • 
Then, Eq. (5) is expressed formally as 

y(k) = H(k)x(k) + w(k) 

in which H(k) is given by 

H(k) = [y(k-l),---,y(k-p),w(*- !),■■■ ,»(*-?)]. 



l P+"/' 



(7) 



(8) 



(9) 



At this stage it should be noted that the matrix H(k) consists of the (unmeasurable) past noise 
sequence {w(-)}. To remedy this inadequate situation, we resort to replace it by 



H(k) = [y(k - 1), • • • ,y(k - p),v m (k - 1), • • • , v m (k - q)] 
in which {v m (-)} is the sequence modified from the innovation sequence v(-) as 

v m (£) = c(£)v(£) (£ = k-q,k-q + l,'-,k-l), 
where 



and 



c(£) 



v(e) = y(e)-H(e)x(e\e-i) 
i + \H(e)p(e\e-i)H T (e) 



(10) 

(ii) 

(12) 
(13) 



Here, x(£\£ — l) and P(£\£ — l) are the one-step prediction and its covariance matrix computed 
by Kalman filter for the past interval. 
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It is a simple exercise to show that the statistical properties of i/ m (-) is the same as that ofw(-), 
i.e., E{v m (k)} = and E{\v m (k)\ 2 } = a 2 (for proof, see Appendix). Then, instead of (8) we 
have the expression, 

y(k) = H(k)x(k) + w(k). (14) 

The procedure for computing H(k) is stated as follows: 

(i) Preliminaries: Assume for the past k{< 0) that {v w (— 1), v m {— 2), • • ',v m {— q)} are set appro- 
priately (may be set all zero), and preassign f(0| — 1), P(0| — 1) and H(0) as initial values. 
Then, at time k (k = 0, 1, 2, • • • ) 

(ii) Computation ofv(£) and c(£): Compute the innovation v(£) and coefficient c(£) by (12) and 
(13) using H(£) = [y(£ - 1), • • -,y(£ - p), v m (£ - 1), • • •, v w (£ - q)]. 

(iii) Computation ofv m (£): Compute v m (£) by (11) using v(£) and c(£) obtained in the previous 
step. 

Repeat Steps (ii) and (iii) for £ = k — q,k — q + 1, • • -,k — 1 to obtain H(k). In computing (12) 
and (13), x(£\£ - 1) and P(£\£ - 1) are computed by the Kalman filter (e.g., Jazwinski, 1970): 

x(£ + l\£) = x(£\£) (15) 

x(£\£) = x(£\£ - 1) + K(£)v(£), (16) 

m - fl W P(/|/-Ww + ^ PMf " 1)AT w (17) 

P(£ + l|£) = P(£|£) + Q (18) 

P(£\£) = P(£\£ - 1) - K(£)H(£)P(£\£ - 1), (19) 

where Q = diag {r^, • • • , T 2 +q }. 

Thus, the estimates of the coefficient parameters {ocj(k)} and {/3y(/c)} are obtained by the 
Kalman filter constructed for (7) and (14) (whose form is the same as (15)-(19) replacing £ 
by the present k). Under the basic assumption that the coefficient parameters vary slowly and 
smoothly, they can be treated like constants in an interval \ around the current time k. Write 
them as &% and /3^ in 4. Replacing the past {w(k — ;')} in (5) by the statistically equivalent 
sequence {v m (k — ;')}, define the sequence y(k) by 

m := y(fc) + E *ikV^ - - E ^«(fc - ;)■ ( 2 o) 

Then, we have the following adequate approximation for (5), 

m = ww pi) 

which implies that the sequence {y(fc)} is stationary because w(k) is the stationary white 



4. Signal Detection 

After obtained the estimates of coefficient parameters, the observation process (14) may be 
written using estimates as 

y(k) = H(k)x(k\k) + w(k) (22) 
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v q 

y{k) + E & ikyfr - = E i^M* - ;) + «>(*)■ ( 23 ) 

i=l ;=1 

Now, let us revive the signal s(k) in the observation data. To do this, replace {y(k)} formally 
b Y {y(k) ~ s ( k )} in ( 23 ) to obtain 



V 
y(k) + Y^& ik y(k-i) 
i=\ 



V 

s(k) + Y^& ik s(k-i) 

i=\ 



+ ^^ jk Vm(k-j)+w(k) (24) 

7=1 



y{k)=Kk) + w{k) f (4) bis 

where y(k) has the same form as (20) and 

s(k)=s(k) + £<x ik s(k-i). (25) 

Note that (4)b is is familiar to us as the mathematical model for the detection problem of signals 
in stationary noise (e.g., Van Trees, 1968). 

Now, consider the binary hypotheses: H 1 : j/(fc) = s(k) + w(k), and H°: y(k) = w(k), and let Y k 
be the stationarized observation data taken up to k, Y k = {y(£),£ = 1,2, •• • ,k}. Since the ad- 
ditive noise w(k) is white Gaussian sequence with zero-mean and variance a 1 , the likelihood- 
ratio function A(k) = plY^H 1 } /Y k \H } is evaluated as follows: 

n(27r) -exp(- ^^ 
Mk) = ^— ; — - ■ (26) 



r y 2 W 

\^1L) z CAJJ S — 
= 1 



^[(2tt) 2 exp 



2(72 
We use rather its logarithmic form, 

L(k):= In A(k) 



Xtmrn-^L^w ( 27 ) 

<r i=1 x> £=1 



as the signal detector. 



5. Simulation Studies 



In this section, we provide a typical set of several simulation results to demonstrate the pro- 
posed method. 

(i) Experiment 1. 
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The top of Fig.l depicts a sample path of the observation data {Y(k)} generated by calculating 
the output of the ARMA(4, l)-model: 

4 

n(k) = -Y^cci(k)n(k-i) + p(k)w(k-l)+w(k). 

z=l 

Time-varying coefficients {oci(k)} and fi(k) are set as 

oci(k) = -1.24sin(0.002fc-0.95), oc 2 (k) = 0.38 - 2cos(0.004/c - 1.89) 
ct3(k)=oc 1 (k), a 4 (fc) = l, j6(fc) = 1.5. 

The bottom of Fig.l shows a signal embedded in the observation data around k = 300 given 
by 

s(£) = l2e- 2 - 78£2 sm(l.26£), 

where £ = k — 300. Figure 2 depicts trend-removed data and stationarized data y{k). The 
trend was removed by setting d = 1. For the Kalman filter (15)~(19), the parameters are set 
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Fig. 1. A sample path of the observation data Y(k) (top) and the embedded signal s(k) (bot- 
tom). 
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Fig. 2. The trend-removed data y(k) (top) and the stationarized observation data y(k) (bot- 
tom). 
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Fig. 3. Log-likelihood function L(k). 

as Q = diag {0.05, 0.05, 0.05, 0.05, 0.05} and a 1 = 40. It should be noted that from Fig. 2 the 
observation data is well stationarized and that even in this figure the signal emerges from the 
background noise. 

Figure 3 shows the result of signal detection by the current log-likelihood ratio function L(k). 
Clearly, it exhibits a salient peak around the true time instant k = 300 and this shows the 
existence of the signal. 



(ii) Experiment 2. 

Efficacy of the signal detector proposed in this paper is also tested for the pulse signal. 
Figure 4 depicts observation data and embedded three pulses. Random noise n(k) is gener- 
ated by the same manner of previous simulation with same coefficients <x,j(k) and fi(k). As a 
signals s(k), a train of pulses with same magnitude is considered: 



s(k) 



20 for D { <k< D f + 5 (f = 1,2,3) 



otherwise, 



where D 1 = 200, D 2 = 500, D 3 = 800. 
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Figure 5 depicts trend-removed data and stationarized data y(k). The trend was also removed 
by setting d = 1. The parameters of Kalman filter are set as the same of previous experiment. 
Figure 6 shows the result of signal detection. Clearly, log-likelihood ratio function L(k) 
has large value around each time when each pulse exists. Thus the signal detection is well 
succeeded. 
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Fig. 4. A sample path of the observation data Y(k) (top) and the pulse signal s(k) (bottom). 
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Fig. 5. The trend-removed data (top) and the stationarized observation data y(k) (bottom). 
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Fig. 6. Log-likelihood function L(k). 

6. Conclusion 

The efficacy of the proposed signal detection method based on the stationarization of 
nonstationary observation data has been confirmed by simulation studies. The key to use the 
Kalman filter to estimate the coefficient parameters of the ARMA noise model is laid on the 
replacement of the unobservable past noise sequence by the equivalent (modified) innovation 
sequence which is observation data-measurable. The stationarization of a nonstationary data 
as introduced in this paper will have potential ability to treat the nonstationary noise or 
observation data in the signal processing. 



Appendix. Proof of Statistical Equivalence Between {w(k)} and {v m (fc)} 
The mean of the modified innovation sequence v m (k) is clearly zero. Indeed, 

£{v m (k)} = c(k)£{v(k)} 

= c(k)£{y(k)-H(k)x(k\k-l)}. 
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Here, recalling that y(k) is given by the form (14), we have 

= c(k) [H(k)S{x(k) - x(k\k - 1)} + S{w(k)}\ 

= c(k)H(k)S{S{x(k) - *(*|* - l)|Yjfc_i}} 

= c(k)H(k)S{8{x(k)\Y k _ 1 } - x(k\k - 1)} 

= 0, 

where Y k _ x = {y(£),0 <£<k-l}. 

Next, the variance of v m (k) is evaluated as follows: 

ntim = c 2 (k)s{v 2 (k)} 

= c 2 (k)S{[H(k)[x(k) - x(k\k - 1)] + w(k)} 2 } 

= c 2 (k) [H(k)S{[x(k) - x{k\k - 1)] [x(k) - x(k\k - l)] T }H T (k) + £{w 2 (k)}] 

= c 2 (k)[H(k)P(k\k-l)H T (k)+(T 2 }. 

If we select c(k) as (13), the variance of v m (k) -sequence becomes a 2 which is just the variance 

o({w(k)}. 

(Q.E.D.) 
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This chapter presents a new framework to design different types of IIR filters based on the 
general technique for maximally flat allpole filter design. The resulting allpole filters have 
some desired characteristics, i.e., desired degree of flatness and group delay, and the desired 
phase response at any prescribed set of frequency points. Those characteristics are important 
to define the corresponding IIR filters. The design includes both real and complex cases. 
In that way we develop a direct design method for linear-phase Butterworth-like filters, using 
the same specification as in traditional analog-based IIR filter design. The design includes the 
design of lowpass filters as well as highpass filters. The designed filters can be either real or 
complex. The design of liner-phase two-band filter banks is also discussed. 
Additionally, we discussed the designs of some special filters such as Butterworth-like filters 
with improved group delay, complex wavelet filters, and fractional Hilbert transformers. 
Finally, we addressed a new design of IIR filters based on three allpass filters. As a result we 
propose a new design of lowpass filters with a desired characteristic based on the complex 
allpole filters. 

Closed form equations for the computation of the filter coefficients are provided. All design 
techniques are illustrated with examples. 

1. Introduction 

The design of allpole filters has been attractive in the last years due to some promising appli- 
cations, like the design of allpass filters (Chan et al., 2005; Lang, 1998; Pun & Chan, 2003; Se- 
lesnick, 1999; Zhang & Iwakura, 1999), the design of orthogonal and biorthogonal IIR wavelet 
filters (Selesnick, 1998; Zhang et al., 2001; 2000; 2006), the design of complex wavelets (Fernan- 
des et al., 2003), the design of half band filters (Zhang & Amaratunga, 2002), the filter bank 
design (Kim & Yoo, 2003; Lee & Yang, 2004; Saramaki & Bregovic, 2002), the fractional delay 
filter design (Laakso et al., 1996), the fractional Hilbert transform (Pei & Wang, 2002), notch 
filters (Joshi & Roy, 1999; Pei & Tseng, 1997; Tseng & Pei, 1998), among others. The majority 
of the methods use some approximation of the desired phase in the least square sense and 
minimax sense. 

The allpole filters with maximally flat phase response characteristic have been specially attrac- 
tive due to promising applications, like the design of IIR filters (Selesnick, 1999), the design 
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of orthogonal and biorthogonal IIR wavelet filters (Selesnick, 1998; Zhang et al., 2001; 2000; 
2006), the design of complex wavelets (Fernandes et al., 2003), the design of half band filters 
(Zhang & Amaratunga, 2002), the fractional delay filter design (Laakso et al., 1996) and the 
fractional Hilbert transform design (Pei & Wang, 2002). 

This chapter presents a new design of real and complex allpole filters with the given phase, 
group delay, and degree of flatness, at any desired set of frequency points. The main moti- 
vation of this work is to get some new promising cases related with the applications of max- 
imally flat allpole filters. In that way, using the proposed extended allpole filter design, we 
introduced some new special cases. 

The rest of the chapter is organized as follows. Section 2 establishes the general equations for 
maximally flat real and complex allpole filters. The discussion of the proposed design is given 
in Section 3 for both, real and complex cases. Different special cases of the general allpole 
filter design is discussed in Section 4. Finally, Section 5 presents some applications of the 
proposed allpole filter design, i.e., linear-phase Butterworth-like filter, Butterworth-like filters 
with improved group delay, complex wavelet filters, fractional Hilbert transformers, and new 
IIR filters based on three allpass filters. 

2. Equations for Maximally Flat Allpole Filter 

We derive here equations for real and complex allpole filters both of order N, delay r, and 
degree of flatness K, at a given set of frequency points. 
We consider that an allpole filter of order N is given by, 

D(z) = ^ (1) 

where a: is a complex constant with unit magnitude, z is the complex variable, and F{z) is a 
polynomial of degree N, 

N 

F(z) = l+^f n z- n . (2) 

n=\ 
In general, the filter coefficients f n , n = 1, . . . , N, are complex, i.e., f n = /r m + ]f\ n where /r w 
and fi n are the real and imaginary parts of f n , respectively. Obviously, if f\ n = 0, we obtain 
real coefficients. 
The phase responses of D(z) and F(z) are related by 

(p D (cv) =^-^W (3) 

where (p^ is the phase of oc, and (pjj (cv) and (Pf(cv) are the phases of D(z) and F(z), respectively. 
The corresponding group delay is the negative derivative of the phase, as shown in (4). 

G(q,) = - %M = %M, (4) 

da) dco 

The conditions for the maximally flat group delay at the desired frequency point cv are 

G{w) = t (5a) 

G^\(v)=0, p = l,...,K, (5b) 

where r is the desired group delay, K is the degree of flatness, and G^\co) indicates the pth 
derivative of G(co). 
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By performing the Fourier transform, equation (2) can be written as 

F(ei") = [F(eJ a; )F*(eJ a; )] 1/2 e^^, (6) 

where F*(e^), is the complex conjugate of F(e^ w ). 

Using (4) and (6) the corresponding group delay G(co) can be expressed as 

G(») = ^ = J^?1, (7) 

v ; do; [ F(e^) J V 

where F^ 1 ) (e^) is the first derivative of F(e^) and ^{-} indicates the imaginary part of {•}. 
Combining (5) and (7), we arrive at 

F (e^) 
F(e^) 






d fc (F( 1 )(eJ a; ) 



da; fc I F(e)°>) 



0, l = l,...,K. (8b) 



The Fourier transform (6) can be rewritten as, 

N 



F(e ]co ) = £) (/ Rw cos(o;n)+/ Iw sin(a;n)) +j £) (/ Iw cos(am) - / Rn sin(am)). (9) 

Substituting (9) into (8), we find that that the conditions given in (8) result in the following set 
of linear equations: 

N 

^ (ft + r) k cos(am + <£ a - <£dM)/rh 
n=l 

N 
+ J^(n + r) k sm(am + (pa -<pD(^))fin = -r k cos((p D (cv) -(pa), k odd, (10a) 

n=l 

N 

J^ (ft + T) fc sin(om + <p a - (p D (^))fRn 

N 

— ^(n + r) k cos (cvn + (p & — (p]j(cv))fi n = T k sin((p]j(co) — (p^, k even. (10b) 

n=l 

Equations (10a) and (10b) are the general set of equations, which includes desired phases, 

group delays and degrees of flatness at given frequency points for both real and complex 

cases. 

Notice that for each frequency point W\, we have K\ + 2 equations (see (10)) and 2N unknown 

coefficients. A consistent set of linear equations (10) is obtained if the following condition is 

satisfied, 

N=(| + l) + (§ + l)+... + (^ + l) / (11) 

where L is the number of frequency points. 
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3. Description and discussion of the proposed allpole filter design 

We describe the design procedure based on general equations for the allpole filter proposed 

in Section 2 

The parameters of the design are the constant oc, the number L, the corresponding frequency 

values co i, I = 1, .. . ,L, phase values <Pd(wi)> I — 1, .. .,L, group delays r (a;/), / = 1, ...,L, 

and degrees of flatness iC/, / = 1, . . . , L. 

For the real case, i.e., f\ n = and a: is a real constant, the relations (10a) and (10b) become 

N 

Y^(n + r) k cos(cvn - (p D (oj))f n = -T fc cos(^ D (o;)), k odd, (12a) 

n=\ 

N 

^(n + r) k sm(con — (p]j(co))f n = r k sin((p]j(co)), k even. (12b) 

n=l 
Similarly, the condition (11), for the real case becomes 

N = (JCi +2) + (K 2 + 2) + • • • + (K L + 2) . (13) 

The algorithm is described in the following steps: 

Step 1. Compute the order of the allpole filter N, using (13) for the real case, and (11) for the 
complex case. 

Step 2. Substitute the frequencies u)\, I = 1, . . . , L, group delays t(coi) and phases (pjj (coj) into 
(12), for the real case, or (10), for the complex case. 

Step 3. Calculate the filter coefficients f n solving the resulting set of equations. 

The following example illustrates the design of real allpole filter D(z), (oc = 1) using three 

desired frequency points, L = 3. 

Example 1. The design parameters are shown in Table 1. 



/ cvi <pp(wi) r(cvi) K t 

1 rc/5 rc/3 3 5 

2 zr/2 7i/4 3 7 

3 4/r/5 n/5 4 4 



Table 1. Design parameters in Example 1, using L = 3 and oc — 1. 

Step 1. From (13), the estimated value of N is 22. 

Step 2. We substitute the frequencies oo\, group delays t(<jj\) and phases §r>(oo\), I = 1, ... ,3 
into (12). 

Step 3. Solving the resulting linear equations, we get the filter coefficients f n . 

Figure la shows the corresponding group delay, while the phase response is presented in 
Fig. lb. The desired phases atcv = n/5, to = tt/2 and to = An/5 are also indicated in Fig. lb. 
The following example illustrates the complex case. 
Example 2. We design the complex allpole filter with characteristics given in Table 2. 

Step 1. The order N of the allpole filter is 13 (see (11)). 
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Fig. 1. Phase response and group delay of the designed real allpole filter in Example 1. 



/ 
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1 7T/3 

2 4tt/5 

3 8tt/5 



7i/6 1/2 8 

-zr/20 1/2 6 
-3/T/20 1/2 6 



Table 2. Design parameters in Example 2. The value L is 3 and cc = 1. 

Step 2. Using (10a) and (10b), we obtain the set of linear equations with 26 unknowns coeffi- 
cients; 13 for /r w and 13 for /j n . 

Step 3. Solving the resulting set of equations, we get the coefficients of the complex allpole 
filter. 
Figure 2 illustrates the phase response and group delay of the designed allpole filter. 
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Fig. 2. Group delay and phase response of the complex allpole filter D(z) in Example 2. 
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3.1 Relationships between allpole filters and allpass filters 

We consider the relations between allpole filters of order N and allpass filters. 
An allpass filter A(z) is related with an allpole filter as follows (Selesnick, 1999), 

V ; D(z) oc*F{z)' 

where D(z) is the paraconjugate of D(z), that is, it is generated by conjugating the coefficients 

of D(z) and by replacing z by z _1 . 

The phase <p A (to) of A(z) can be expressed as 

<p A (w) = -wN + 2<p D (w), (15) 

where the desired phase (pjj (co) is given by 

* D («) = tA^E. (16) 

From (15), the group delay of the complex allpass filter t a (co) is given by 

t a (co) = N + 2t(co), (17) 

where t(co) is the group delay of D(z). 
Using (17), it follows 

r(o0 = ^lM^. (18) 

It is well known that the structures based on allpass filters exhibit a low sensitivity to the filter 
quantization and a low noise level (Mitra, 2005). Therefore, the relationship (14), between all- 
pass and allpole filters, gives the possibility to use efficient allpass structures in the proposed 
design. 

4. Promising special cases 

The proposed allpole filters have desired phases, group delays and degrees of flatness at a 
specified set of frequency points. In this section we introduce some new special cases of 
the proposed design (10), which are used for the design of complex allpole filters, complex 
wavelet filters, and linear-phase IIR filters. 

4.1 First order allpole filters 

Using (12), the filter coefficient /ri is computed as follows: 

_ sin(</> Dl ) 

/ri - sin^-^r (19) 

where (p^ is the desired phase at co = co\. 

To ensure the stability of the allpole filter, we have 

/~ n 1 — cos(2o;i) /rk ^. 

tan( ^ ) > sin^) ■ (20) 
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Similarly for the complex case, the filter coefficient f\ is 

, sin(</>„ - 0p 2 )eJ ( " 1+ *'-*°i ) - sin(^ - ^eK^*.-^) 

/l = '• — 7 \ / V^-U 

sm(cv 1 -cv 2 + (pD 2 -<pDi) 

where (p^ and cpjj 2 are the phases of the allpole filter at the desired frequency points co = co\ 
and co = co 2 , respectively The stability of the allpole filter is satisfied if the following equation 
holds 

. ,. . v . cos(o; 1 -(v 1 + <pcL-<p Dl )-\ cos(fa - ^) | 

tan((pD 7 — (poc) < — : — 7 s ^—/ v— • (22) 

sin(o;i -<v 2 + <Pcl-<I>d 1 )+ sin(^Di - <pu) 

4.2 Second order allpole filter 

We consider the following two cases. 

Case 1. For co = co\, we specify the desired phase <pv x and group delay r. Substituting these 

conditions into the general equations (12), the resulting filter coefficients are 

(t + 1) sin(2o;i) - sin(2o;i - 2» Dl ) 
/R1 (t + 1) sin co\ - sin(o;i - <f Di ) cos(2o;i - <p Dl ) ' 

= t sin q;i + sin(ft Dl ) cos(o; 1 - <p Dl ) 

7R2 (t + 1) sin a^ - sin(a;i - (p Dl ) cos(2co 1 - <p Dl )' 

Additionally, the condition for the stability of the allpole filter is 

T> _ i+ |sin(2a; 1 -2 to )| 
2sino;i 

Case 2. For two phases (p^ and (pjj 2 at the frequencies tO\ and co 2/ the filter coefficients are 

= sin(2o;i - <p Dl ) sin(ftp 2 ) - sin((p Dl ) sin{2co 2 - (pp 2 ) 

sin(co 2 ~ <Pd 2 ) sin(2o;i - (p Dl ) - sin(co 1 - (p Dl ) sin(2co 2 ~ <pD 2 )' 
f = sin((p Dl ) sin(co 2 - <pp 2 ) - sin(a^ - <p Dl ) sin(ftp 2 ) 

lR2 sin(co 2 - (p Dl ) sin(2a;i - (p Dl ) - sin(a;i - (p Dl )sm(2co 2 - <pD 2 Y 

Furthermore, the stability of the allpole filter is guaranteed if the equation 

sina^sina^tan^-fe) , nQ . 

tan a;! - <p D ) < — ? , (28) 

cos a?i cos cc»2 — 1 + | cos co\ — cos a^2 1 

is satisfied. 

4.3 Complex Thiran allpole filters 

We generalize the result proposed by Thiran (Thiran, 1971), for the design of real allpole filters 
that are maximally flat at co = 0, to include both the real and complex cases. The required 
design specifications are the order of the allpole filter N, group delay r(co) at co = 0, tq, 
degree of flatness K, and the phase value (p a . 
Consequently, the allpole filter must satisfy: 

A.l The degree of flatness at co = is K, where K can be either 2N — 2 or 2N — 3. 
A.2 The phase value (pjj (co) is equal to zero at co = 0. 
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4.3.1 Degree of flatness K = 2N -2 

Substituting conditions AX and A.2 into the set of equations (10), we compute the complex 

coefficients as follows 



„ (N\ 2(2t + 1)„ 



f'--^yi:)i^m-A r ' + ^-" m ^')' < 2 " 



where n = 1, . . . , N, the binomial coefficient is given by 



n) n\(N — n)\' 



(30) 



and the Pochhammer symbol (x) m indicates the rising factorial of x, which is defined as (An- 
drews, 1998), 

, x f(*)(* + l)(* + 2)...(* + m-l) m>0, 

m \1 m = 0. 

The expression in (29) is the extension of the result proposed in (Thiran, 1971), which includes 
both real and complex cases. If (p a is or re, the imaginary coefficients are zero, and the result 
is a real allpole filter, consistent with (Thiran, 1971). For cp^ = ±tt/2, the filter is a real allpole 
filter (this case is not included in (Thiran, 1971)). For all other phase values, the imaginary 
coefficients are strictly non-zero, i.e., the filter is complex. 

4.3.2 Degree of flatness K = 2N - 3 

In this case, in order to get a degree of flatness K = 2N — 3, we set fjj^f = 0. Consequently, the 

filter coefficients are 

_ (N\ 2(2t + 1) w _ 1 / (»-N)e^cos^ \ 

fn ~ ( 1} {n)(2r + N + l) n { ro + n + n 2r + N )' (32) 

where n = 0, . . . , N. 

In contrast with (32), to obtain a different solution, we now set /r^ = 0. Therefore, we have 

(N\ 2(2t + !)„_! / neifr / (N - h)(t + Ncos 2 ^) 

(33) 
where n = 0, . . . , N. 

We illustrate the design with one example. 

Example 3. The desired phase (p a/ and the group delay tq at co = 0, are —n/6, and 7/3, 
respectively. The order N of the filter is 5. 

We compute the corresponding filter coefficients using (29), (32), and (33). The resulting group 
delays of D(z) are shown in Fig. 3a, while the phase responses of the designed filters are 
shown in Fig. 3b. 

4.4 Complex allpole filter with flatness at w — and w — n 

Now, we present the design of complex allpole filters of order N (any positive integer) with 

flatness at cv = and co = n. 

The design conditions are: (More detailed explanation is given in Section 5.1.) 

B.l The phase response of D(z) is flat at the frequency points cv = and co = n with group 
delays t(0) = t(tt) = -N/2. 
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Fig. 3. Group delays and phase responses of the complex allpole filters in Example 3. 

8.2 The degree of flatness at these frequency points is the same, i.e., K = N — 2. 

B3 The phase values of the allpole filter (po{w) at cv = and cv = n, are and n(2N + 
(21 + 1)) /4, respectively, where I is an integer. 

BA The desired phase value (pjj (cv) at the given frequency to = cop is cpp, i.e., cpp = (pjj (cvp)- 

Substituting conditions B.l-BA into (10a) and (10b) and solving the resulting set of linear 
equations, we arrive at 



fn 



^(V2eJ(^ + *)-j) n 



odd, 



where 



-J 



(-1) 



[N/2] 



cot 



(pp 



cVpN 



1 tan ] 



(?) 



(34) 



(35) 



and /{•} indicates the angle of {•}, while ["•] stands for the floor function. 

Next example illustrates the proposed design where the parameters of the design are the filter 

order N and the phase value cpp at the frequency point ojp. 

Example 4. We design a complex allpole filter using the following specifications: the order of 

the allpole filter is N = 7 and the phase value (pjj (cv) at cvp is 1.2tt, where cop = 0.3 n. 

The group delay and phase response of the designed filter are presented in Fig. 4a and 4b, 

respectively. 

4.4.1 Closed form equations for the singularities of the allpole filter 

In the following, we consider the computation of the poles of D(z). 

Using (34), we obtain the z-transform of the denominator of D(z) defined in (1) as, 



m 



n ' n odd ^ n ■ 



(36) 
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Fig. 4. Group delay and phase response and of the complex allpole filter in Example 4. 



After some computations, we get 



F(z) 



V2 



(cos (pot — sin^ A; )(l + z 1 ) N — (j — 1) sin</> a (l — z 



1\N 



Therefore, the corresponding poles are 



Vk 



7fc + l 
7k -^ 



where k = 0, . . . , N — 1, and 



(37) 



(38) 



Ik 



V2 



1 -COtd> K 



e > 4n 



(39) 



4.5 Complex allpole filters with flatness at cv — 0, and to — ±o; r 

In this section, we design a complex allpole filter with the following characteristics: 

C.l The order N is even. 

C.2 The allpole filter has flat group delay at the frequency points co = 0, cv = —co v , and 
co = co r . The degrees of flatness are K x (w = 0) = N - 2, K 2 (co = ±co r ) = N/2 - 2. The 
group delay at those frequency points is r(0) = t(±co t ) = —N/2. 

C.3 The desired allpole phase value <Pd(lo) at the given frequency co = co p is (p p/ i.e., ^p = 

CA The phase values of the allpole filter (p£){co) at co = 0, co = —co V/ and co = co r are 0, 

tt/3 + co r N/2, and n/3 - co r N/2, respectively. 

Substituting conditions C.l-CA into (10a) and (10b) and solving the resulting set of linear 
equations, we have 



fn = ("I)" 



N\ 4e#« fN/2\ 

'~7T\ n ) c N / «(w r )cos(^„ + 7r/6) 



(40) 
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where n = 0, . . . , N/2, 



(pcc = ^ { V3R p cot(>p - cv p N/2) + 1 + j V3(R p + 1) } ; 



and 



where 



N-l sin Nr^E 



R n 



Cn K«f 



P c N/N/2 (o; r )+2C N (w r/ Wp)' 

N/2-1 



(41) 



(42) 



£ (.^N/a+n/^/ j CNM(a ; r ) cos ( (N /2- M ) Wp ). (43) 

The function c^ /n (cv Y ) for different values of N is given in Table 3. Moreover, we have 
CN,o(<^r) = and f n = f N _ n . 

Example 5. The desired design specification is as follows: the allpole filter order is equal to 8, 
07p = 0.35ti, co r = 0.75n, and cp p = 1.5tt. The resulting group delay and phase response of 
the designed filter are shown in Fig. 5. 

Group delay Phase response 




0.25 0.5 0.75 

Normalized frequency 

(a) 




0.25 0.5 0.75 

Normalized frequency 

(b) 



Fig. 5. Group delay and phase response and of the designed complex allpole filter in Exam- 
ple 5. 



5. Design of MR filters based on allpole filters 

5.1 Direct design of linear-phase MR Butterworth filters 

A filter H(z) has linear-phase if, 



H(z) 



K H(z), 



(44) 



where H(z) is not necessary causal, z~ k is the delay, the complex constant c has unit magni- 
tude and H(z) is the paraconjugate of H{z), that is, it is generated by conjugating the coeffi- 
cients of H(z) and by replacing z by z _1 . 

It has been shown that causal Finite Impulse Response (FIR) filters can be designed to have 
linear-phase. However, Infinite Impulse Response (IIR) filters can have linear-phase property 
only in the noncausal case (Vaidyanathan & Chen, 1998), (the phase response is either zero or 
n). It has been recently shown that filters with the linear-phase property are useful in the filter 
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N 



12 



16 



1 

2 

10 3 

4 

5 



CtynO^r) — CN,N-n(toi) 



- cos(o; r ) 



1 — cos(o; r 
1 — cos(2o; r 



10- 



1 — cos(o; r 

1 — cos (2a; r 

)cos(o; r ) — cos(3o; r 



1 — cos(o; r 

1 — cos(2o; r 

7 — 6cos(a? r ) — cos(3o; r 

17 - 16cos(2o; r ) - cos(4o; r 



126 



11 

100cos(o; r ) 



1 — cos(o; r 

1 — cos (2a; r 

6 — 5cos(o? r ) — cos(3o; r 

— 10cos(2o; r ) — cos(4o; r 

— 25 cos(3o; r ) — cos(5o; r 



11/2 
c 

66 — 50cos(o; r ) 
262-225cos(2o; r ) 



1 — cos(o; r 
1 — cos(2o; r 

- 9/2cos(o; r ) — cos(3o; r 

- 8cos(2o; r ) — cos(4o; r 

- 15cos(3o; r ) — cos(5o; r 

- 36cos(4o; r ) — cos(6o; r 



1 1 — cos(o; r 

2 1 - cos(2o; r 

3 26/5 - 21/5 cos(o; r ) -cos(3o; r 
14 4 8 - 7 cos (2co r ) - cos (4o; r 

5 143/3 - 35 cos(o; r ) - 35/3 cos(3o; r ) - cos(5o; r 

6 127 - 105 cos(2o; r ) - 21 cos(4o; r ) - cos(6o; r 

7 1761 - 1225 cos(o; r ) - 441 cos(3o; r ) - 49 cos(5o; r ) - cos(7o; r 



1 — cos(o; r 

1 — cos(2o; r 

5 — 4cos(o; r ) — cos(3a? r 

37/5 - 32/5 cos(2o; r ) - cos(4o; r 

39 — 28cos(o; r ) — 10cos(3o; r ) — cos(5o; r 

87 - 70cos(2o; r ) - 16cos(4o; r ) - cos(6o; r 

715 - 490cos(o; r ) - 196cos(3o; r ) - 28cos(5o; r ) - cos(7o; r 

3985 - 3136cos(2o; r ) - 784cos(4o; r ) - 64cos(6o; r ) - cos(8o; r 



Table 3. Function c^ /n (co T ) for different values of N. 



bank design and the Nyquist filter design and different methods have been proposed for this 
design (Djokic et al, 1998; Powell & Chau, 1991; Surma-aho & Saramaki, 1999). 
A linear-phase lowpass IIR filter H(z) can be expressed in terms of complex allpass filters as 
(Zhang et al, 2001), 

H{z) = 1 -[A{z)+A{z)], (45) 

where A(z) is a complex allpass of order N (see (14)). 

We can note that the filter defined in (45) satisfies the relation (44) if k = and c = 1. 
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The main goal is to propose a new technique to design real and complex IIR filters with linear- 
phase, based on general design of Section 3, where the design specification is same as in tra- 
ditional IIR filters design based on analog filters, i.e., the passband and stopband frequencies, 
cop and co s , the passband droop A p , and the stopband attenuation A S/ shown in Fig. 6. 
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Fig. 6. Design parameters for low pass filter. 



We relate the design of linear-phase IIR filter with allpass filter and in the next section we use 

the general approach to design the corresponding allpole filter. 

First, we establish the conditions which the auxiliary complex allpass filters from (45) has to 

satisfy. 

From (45), the magnitude response of H(z) can be expressed as, 



|H(e^)| = |cos(<£ A (a;))|, for all a;. 



(46) 



The magnitude responses of \H(e^) | at co = 0, and co = re are 1 and 0, respectively (see Fig. 6). 
Therefore, the values of <Pa(lo) at these frequency points are and (11 + 1)tt/2, respectively, 
where I is an integer. Since the magnitude response of H(z) decreases monotonically, relation 
(46) can be rewritten as, 



|H(e^)| = cos ((p A (u)), < a; < re. 



(47) 



Note that \H(e^ C0 )\ has a flat magnitude response at co = and co = re, and that the filter A(z) 
has a flat phase response at the same frequency points. As a consequence, the corresponding 
group delays Ta(0) and t a (ti) are equal to 0. 
Considering the value A p in dB we write 



201og 10 |H(e 



jon 



-A p . 



(48) 



From (47) it follows, 

<P P A = <M^ P ) = arccos (l(T A p /20 ) . (49) 

In summary, the conditions that the auxiliary complex allpass filter in (45) needs to satisfy are 
the following: 

V.l The phase values of (Pa{lo) at co = and co = re are and (11 + l)re/l, respectively. 

V.l The phase response of A(z) is flat at co = and co = re. Therefore, t^(0) = t a (tt) =0. 
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V3 The phase value (p p ^ is controlled by A p (see (49)). 

In the following, we use the results from Section 3.1 and the Conditions V.1-V3 in order to 
obtain the corresponding conditions for the allpole filter D(z). 

5.1.1 Design of flat linear-phase MR filters based on complex allpole filters 

We relate the allpass filter from (45) with the corresponding allpole filter. 

Using (16) and the phase values <Pa(w) at co = and co = n (see Condition V.Y), we get 

0(0) =0 and (p(n) = n(2N + (21 + l))/4. 

Now, from (18) and Condition V.2, we have t(0) = t(tt) = -N/2. 

Finally, the following relation is obtained using Condition V3 and (16), 

arccos (l(T a p /20 ) + co p N 
(p D (co p ) =cp p = ^ '- . (50) 

As a consequence, the corresponding conditions that the allpole filter D(z) has to satisfy are: 

£.1 The phase values of D(z) at co = and co = n are and n(2N + (21 + l))/4, respec- 
tively. 

£.2 The group delay r(co) of D(z) at co = and co = n are —N/2. 

£3 The phase value of D(z) at co p , (pofap), is given by (50). 

For a filter having coefficients given in (34) the Conditions £.1 and £.2 are satisfied. 
From the Condition £3 and (35), the corresponding value of (p^ (N, co p , A p ) is equal to 

^ a (N,a; p ,A p ) = z{-j-l-(-l)r N / 2 lVtan N (^)}, (51) 

where 



/ 10 A p /20 , i 

We note that the resulting allpole filter has a causal and an anticausal parts. The causal part 
can be implemented with the well known structures for allpass filters while the anticausal 
part can be implemented with the structures proposed in (Vaidy ana than & Chen, 1998). 
The degree of flatness of the allpass filter A(z) at co = and co = n is equal to N — 2. Based 
on this result it can be shown that we have 2N — 1 null derivatives in the square magnitude 
response |H(e^)| 2 at co = and co = n. 

5.1.2 Closed form equations for the singularities of H(z) 

It follows from (37) and (45) that the transfer function H(z) is given as, 

H(z) = (1+ !" 1)N 5 Z) , (53) 

V ; 2z- N F(z)F(z) 

where 

E(z) = (1 - sin(20«))(l +z" 1 ) N + ((j + 1) - (j - 1)) 8in0«(cos(0«) - sin(tf> a ))(l - z^f . 

(54) 
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We note that the transfer function H(z) has N zeros at z = — 1 and the other zeros are at (see 
(54)), 



(56) 



k Ak-i' 

where k = 0, . . . , N — 1, and the parameter /3 fc is given by, 

2i — • J e^N* N even, 

V i-sin(2^ a )y 

[2-. — . )rZ I e ] 2N n N odd. 

V l-sin(20 a ) 7 

It is easily shown that the absolute values of z^- in (55), for even values of N, are always 
different than 1. However, there also exists one absolute value of z^, for N odd, which is equal 
to 1, i.e., there is a zero on the unit circle. The corresponding frequency coq is expressed as, 

/ l-cos(2d> a )\ 2N 

cv = n + 2 arctan 2- ^-^( . (57) 

V 1 -sin(2<p a ) / 

As a consequence, the frequency at which H(e^) is equal to —1 is given by 

a;! = 7i + 2arctan -^- . 58 

\2 1 -sin(2<p a ) / 

Finally, the transfer function H(z) has 2N poles which are poles of the corresponding complex 
allpole filters D(z) and D(z) (see Section 4.4.1). 

5.1.3 Description of the algorithm 

The proposed algorithm is described in the following steps: 

Step 1. Estimate the order of the allpole filter using the following equation, which can be 
obtained by solving (p a (N, a)p, A p ) = (j>cc(N, co S/ A s ), 



N ■ 



lQ gio(^f) 



1Q A p /20 + 1 1QA/2Q + 1 



where |_- L i s the ceiling function. 
Step 2. From the values N, cvp and A p , compute the phase value <p a (N, o; S/ A s ), using (51). 
Step 3. Using (34), compute the filter coefficients f n . 
Step 4. Calculate the filter coefficients of H(z) using (45). 
We illustrate the procedure with the following example. 

Example 6. We design the IIR linear-phase lowpass filter with the passband and stopband 
frequencies cVp = 0.25 n and cv s = 0.5 n, respectively. The passband droop is A p = 1 dB, while 
the stopband attenuation is A s = 65 dB. 

Step 1. Using (59), we estimate N = 10. As a consequence, the filter H(z) is real. 
Step 2. We calculate the phase value <p a (N, <v S/ A s ), to be fa (N, to s , A s ) = -0.749925tt. 
Step 3. The filter coefficients f n are computed from (34). 
Step 4. We compute the coefficients of the designed filter H(z). The magnitude response of 

the designed filter is given in Fig. 7. 
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Fig. 7. Example 6. 











5.1.4 Linear-phase MR highpass filter design 

Now, we extend the proposed algorithm for lowpass filter to highpass filter design. 

Using the power-complementary property (Vaidyanathan et al., 1987), it can be shown that 

the corresponding complementary filter of H(z), defined in (45), is given by 



H 1 (z) = ^[A(z)-A(z) 



(60) 



where Hj (z) is a highpass filter. 

Using (60), the phase value cp pA ' s expressed as, 

<t> pA = arcsin (lO~ A p /20 ) . 
Similarly, the phase value (p a (N, u>p, A p ) is given by, 



p x (N,to p ,Ap) = Z 



-(j + l)cot-(^ 



o; p n 2 (-l)LN/2J 



A c 



(61) 



(62) 



Finally, the filter coefficients of Hi (z) are computed using (60). 

The following example illustrates the procedure. 

Example 7.The parameters of the design of the highpass filter are: the passband and stopband 

frequencies are cv p = 0. 75 n and co s = 0.4 n, respectively. The stopband attenuation and 

passband droop are 50 dB and 1 dB, respectively. 

The resulting filter order is equal to 6 and (p a (N,cVp, A p ) = — 0.002569tt. The magnitude 

response, the passband and stopband details of the designed filter are shown in Fig. 8. 

5.2 Direct design of linear-phase MR filter banks 

The modified two-band filter bank (Galand & Nussbaumer, 1984), is shown in Fig. 9. The 
analysis filter Hq(z) and the synthesis filter Gq(z) are lowpass filters, while the analysis filter 
Hi (z) and the synthesis filter Gi (z) are highpass filters. However, both the analysis and the 
synthesis filters are not causal. As a difference with traditional structure, in this structure 
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Fig. 8. Magnitude response of H\ (2) in Example 7. 
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Fig. 9. Modified two-band filter bank. 



there are two extra delays, one in the highpass analysis filter and another one in the lowpass 

synthesis filter (see Fig. 9). 

The output Y(z) is obtained using some multirate computations (Jovanovic-Dolecek, 2002), 

i.e., 

Y(z) = ^(x(z)(c*(z)Ho(z)^ 

(63) 
The output of the filter bank (63) suffers from three types of errors, i.e., aliasing, amplitude 
distortion and phase distortion. 

To avoid aliasing, the synthesis filters are related to the analysis filter Hq(z) in the following 
form (Vaidyanathan et al, 1987), 



G (z)=H (z) / G 1 (z) = H (-z) 



(64) 



where Hq(z) is the paraconjugate of H$(z) and Hi(z) = Hq(—z). 

The amplitude and phase distortions are eliminated if the analysis filters are chosen to satisfy 



H (z)HoW + Ho(-z)H (-2) = l. 
From (65), the following relation holds, 

|H (e^)| 2 + |H (e)^-' I ))| 2 = l. 



(65) 



(66) 
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The relationship between the passband frequency co p and stopband frequency co s of Hq(z), is 
given by, 

(Vp + co s = n. (67) 

Additionally, using (66) and (67) we have 

10 -A p /10 + 10 -A s /10 = 1/ (6g) 

where A p and A s are the passband droop and the stopband attenuation in dB. 
According to (Zhang et al., 2001), the analysis filters are given by, 

H (z) = i[A(z)+A(z)], (69) 

H 1 (z) = ±[A(z)-A(z)}, (70) 

where A(z) is a complex allpass filter and A(z) is its paraconjugate. 

From (69) and (70), we can see that the design of perfect reconstruction filter banks is reduced 

to the design of the complex allpass filter A(z). In the following, we present one method for 

the modified two-band filter bank design based on the results obtained in Section 5.1. 

The perfect reconstruction condition for the modified two-band IIR filter banks is established 

in (Vaidyanathan et al., 1987; Zhang et al., 2001), which implies that the poles of Hq(z) and 

Hi(z) must appear on the imaginary axis and in pairs ]p and l/]p, where p is a pole. From 

this condition, it follows that the filter coefficients given in (34) must be imaginary for even 

values of n (Vaidyanathan et al., 1987). 

Consequently, the values of <p a in (34) for an even N, must be 



-g/r for y even, 

3 
Similarly, the values of <p u when N is odd must be, 

\ re for ^^ even, 

i± 

2 



3 -n for #odd. (71) 



7 n for N+1 odd . W 



5.2.1 Description of the algorithm 

In the following, we describe the proposed algorithm for a linear-phase IIR filter banks. The 
IIR filters are real if N is even, otherwise they are complex. 
The steps of the algorithm are described in the following 

Step 1. Calculate the order N of the allpole filter using (68), (67) and (59). (Note that the filter 
H (z) has order 2N.) 

Step 2. If N is even compute the filter coefficients (34) using (71), otherwise use (72). 

We illustrate the method with the following examples. 

Example 8. Stopband frequency co s of the analysis filter Hq(z) is 0.65zr, while the stopband 

attenuation A s is 45 dB. 

Step 1. From (68) and (67), it follows that A p = 1.373381 x 10" 4 and cv p = 0.35 n. Using (59), 
the order of the complex allpole filter is 12. From (71), (p a = —\n. 
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Step 2. We compute the filter coefficients using (34) and (71). 
Figure 10a shows the corresponding magnitude responses of Hq(z) and Hi (z). 

The following example illustrates the design of IIR filter banks using complex IIR filters. Ex- 
ample 9. We design the IIR filter bank with the following specifications co s = 0.75zr and 
A s = 60 dB for the analysis filter H (z). 

Step 1. The estimated value of N is 9 since cv p = 0.25tt and A p = 4.342946 x 10" 6 . From (72), 

(pa = -\n. 
Step 2. Using (34) and (71), we compute the allpole filter coefficients. 

The magnitude responses of Hq (z) and Hi (z) are shown in Fig. 10b. From Fig. 10b, (57), and 

(58), we note that both filters Hq(z) and Hi(z) have notch frequencies at coq = 1.512254tt 

and a>i = 1.48774571, respectively. 

In general, for N odd, both analysis filters have notch frequencies in the vicinity of to = 

3tt/2. 
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Fig. 10. Magnitude responses of the analysis filters in Examples 8 and 9. 



5.3 Butterworth filters with an improved group delay 
5.3.1 Linear-phase Butterworth filters 

We relate the linear-phase Butterworth filter given in Section 5.1 with the corresponding stable 

and causal IIR filter. 

We remember that a linear phase filter H(z) can be expressed as 



H(z) = cz- k H(z), 



(73) 



where z~ k is the delay, c is a rear or complex constant with unit magnitude and H(z) is the 

paraconjugate of H(z). 

Using k = and c = 1, the linear-phase IIR filter H(z) can be expressed as (Powell & Chau, 

1991), 

H{z) = H c {z)H c {z). (74) 

where H c (z) is a causal and stable IIR filter. Consequently, the corresponding Fourier trans- 
form H(e^ w ) is real and positive for all to. 
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We note that the linear-phase filter H(e^) given in (53) takes both positive and negative val- 
ues for N odd (see (58)). However, when N is even, the function H(e^) take only positives 
values. Therefore, the condition (74) is satisfied. 
In the following we design the filter H c c(z) from (53). 

From (53)-(54), it is easily shown that the polynomials E(z) and F(z) are symmetric for even 
values of N. Consequently, they can be expressed as, 

E(z)=z- N/2 E (z)E (z- 1 ), (75) 

F( 2 )= 2 - N/2 F (z)F (2- 1 ), (76) 

where Eq(z) and F$(z) are subfilters of E(z) and F(z), respectively. 

From (75) and (76), the transfer function H(z) (see (83)) can be rewritten as 

= z-^l + z-^E 0{ z)E^) 
2z-NF (z)F (z)F ( 2 -i)F ( z -i) 



Using (77) and (74), it follows 



H c (z) = ( 1 + 2 ' 1 )! /2 W. (78) 

^Fo( 2 )F ( 2 -i) 



Therefore, the transfer function H c (z) has N/2 zeros at z = — 1. 

There exist many polynomials Eo( z ) an d fo( z ) satisfying (78). In order that H c (z) be stable, 
all zeros of F$(z) must be inside the unit circle. Moreover, it can be shown that for the given 
value of N, there exists 

N poly =2^ 4 J, (79) 

polynomials Eq(z). 

In the following example we design IIR filter H(z) with linear-phase and the corresponding 

filter H c (z) using (77) and (78). 

Example 10. We design an IIR filter H(z) with the passband frequency, co p = 0.3/1, and 

passband droop, A p = 1 dB. The allpole filter order is 8. 

Consequently, H(z) has 8 zeros at z = — 1, while the remaining zeros z^, k = 0, . . .,7, and 

poles pfo k = 0, . . . , 2N — 1, are calculated using (55) and (38), respectively. 

According to (79), there are four different polynomials for Eq(z). The zeros of the first polino- 

mial Eq ' (z) are zq,Zq,Zi, and z\. Similarly, we can obtain the polynomials Eq (z), I = 2,3,4, 
which shown in Table 4. 





z 0l 


,z 7 - 


= l/z 3 =zj 


Zi,Z6 = 1/Z2 = Z^ 


Z2,Z5 = 1/Zi = Z| 


23/ 24 = 1/Z = Z\ 


E£\z) 






X 


X 






Zf\z) 






X 




X 




4 3) ( Z ) 








X 




X 


4 4) ( 2 ) 










X 


X 



Table 4. Different polynomials for Eq(z). 

The group delays of H c (z) for all Eq ^ (z), / = 1, . . . , 4, are shown in Fig. 11. 
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Fig. 11. Different group delays for the IIR filter H c (z) in Example 10. 



We have different degrees of nonlinearity as illustrated in Fig. 11 for different E Q 






:(2), 



Z = l,... / 4. 

In this example, the group delay for Eq L) (z) is more linear than the others. Therefore, for this 

(2) 

example the best polynomial is Eq ' (z). 

The next issue is how to select in general case the best polynomial for Eq(z). 

Numerous examples indicate that it is necessary to satisfy the following two conditions: 

1. The number of zeros of H c (z) inside and outside the unit circle, N\ and N 0/ respectively, 
are related as 

Ni > N , 



where 



N 



if y has the same parity of N, 



I"t1 ~ 1 if I"t1 ~ 1 has the same P arit y of N - 

2. For each zero z m inside, and each zero Z\ outside of the unit circle, we have 

1 



< z n 



(80) 



(81) 



(82) 



5.3.2 Description of the algorithm 

The design parameters of the algorithm are passband and stopband frequencies, co p and cv s , 
respectively, the passband droop A p and the stopband attenuation A s . 
The algorithm has the following design steps: 

Step 1. We estimate the order of the allpole filter D(z) using results from Section 5.1, 



N ■ 



lQ gio (at 7 
>gio(^ 



A. 



loA/io + i 



Ap/10 . 



IO^p 



1 



A" 



lO^s/io + i 
10 A s/io _ i 



(83) 



If the estimation filter order N is odd, increase it by one. 
Step 2. Using the estimated value of N, we calculate the value of the phase <p u (N, a)p, A p ) as, 

<p a (N,co p , A v ) = Z {-j - 1 - (-1) N / 2 tan N (^) A p "} . (84) 
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Step 3. Compute poles and zeros of H c (z) as indicated in the following 

1. The zeros and poles of H(z) are obtained using (55), (56), (38) and (39), respectively. 

2. The N poles of H(z), which are inside the unit circle, become poles of H c (z). 

3. We select N/2 zeros of H(z) which satisfy conditions (80)-(82) to be zeros of the filter 
H c (z) and the others N/2 zeros are at z = — 1. 

Step 4. Using the MATLAB function poly . m, we find the transfer function H c (z). 

Example 11. We design the IIR filter with the following specifications: the passband and 
stopband frequencies are 0.25 n and 0.55 n, respectively; the passband droop and stopband 
attenuation are 1 dB and 50 dB, respectively. 

Step 1. From (83), it follows that N = 12. 

Step 2. Using the estimated value N and (84) we have, (/> a (N, top, A p ) = —0.75n. 

Step 3. The resulting pole-zero pattern of H c (z) is shown in Fig. 12a. 

Step 4. We compute the filter coefficients of the transfer function H c (z). 
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The group delay of the designed filter is shown in Fig. 12b, while Figs. 12c and 12d present 
the magnitude response. 

We compare our result with the traditional IIR Butterworth filter using the following specifi- 
cation: the filter order is equal to 12 and cv c = 0.2689 n. Figure 12b and 12c show the group 
delays and magnitude responses of the Butterworth filter and the proposed one. 
Notice that the proposed filter H c (z) has a better group delay than the traditional Butterworth 
filter. 

5.4 Complex wavelet IIR filters 

The main idea is to generalize the design of real IIR wavelets presented in (Phoong et al., 1995; 
Selesnick, 1998) and (Zhang et al., 2006) in a way that the complex case is also included. To 
this end we use the general approach for the complex allpole filter design from Section 4.3. 
We generalize the result in (Selesnick, 1998), replacing the real allpass filter A(z~ 2 ) with the 
complex allpass filter A(z 2 ), i.e., 

H (z) = 1 -[a(z 2 )+z- 2M+1 A(z 2 )}, (85) 

Hj (z) = \ [a(z 2 ) - z- 2M+1 A{z 2 )] , (86) 

where Hq(z) and H\ (z) are lowpass and highpass filters, respectively, and M is arbitrary inte- 
ger. 
Knowing that A(z) = 1/ A(z) (see (14)), it is easy to verify that Hrj(z) can be rewritten as, 



H (z) = ^p[z- k + A 2 (z 2 )} / (87) 



Mz 2 ) L- 
n ow = 

where 

k = 2M - 1. (88) 

Now, the problem to design complex filters (85) and (86) is reduced to the design of an allpass 
filter A{z), which has the phase response equal to — kcv/4 near to to = 0. 
The group delay of A{z) at cv = is equal to fc/4. Then, the corresponding group delay t^(0) 
of A(z) is expressed as, 

2M - 1 
tao = — £— , (89) 

where tao = Ta(0). 

Using (17), the corresponding group delay of the complex allpole filter D(z) is 

2M - 4N - 1 
to = g • (90) 

The design of biorthogonal wavelet filter based on real allpass filter is proposed in (Phoong 
et al., 1995). The generalization of this result is written in the form 

H (z) = \[z- 2M +z- 1 A(z 2 )}, (91) 

H X ( Z ) = -A(z 2 )H (z) +z- 4M+ \ (92) 

where A(z) is a complex allpass filter and M is any integer. 



298 Signal Processing 



In this case, we first design the corresponding allpass filter A(z) having the group delay at 
co = equal to, 

r M =™f±. (93) 

The corresponding group delay Tq of D(z) is given by, (See (17)), 

2M-2N-1 
to = ^ • (94) 

Finally, the generalization of the orthogonal wavelet filters proposed in (Zhang et al., 2006) is 
given as, 

H (z) = 1 -[l+z- 2M +^A(z 2 )} r (95) 

H 1 (z) = \[z- 1 -z 2M A(z 2 )}. (96) 

The group delays of the complex allpass filter A(z) and allpole filter D(z) at co = are the 
same as for the filter (91) (see (93) and (94)). 

We design the complex allpass filter A(z) using complex Thiran allpole filter D(z) given in 
Section 4.3, i.e., 

() D(z) e-iP- l+/ l2 -i + ...+/ NZ -N • (y/; 

We can notice that by setting different values of the phase <p a of the corresponding complex 

allpole filter D(z), we can obtain different types of complex wavelet filters. 

The following example illustrates the proposed method. 

Example 12. We consider the design of complex wavelet filters using the methods proposed in 

(Selesnick, 1998), (Phoong et al., 1995) and (Zhang et al., 2006). We design a complex allpass 

filter of order N = 6, the phase value at co = is equal to <p u = —n/5, and delay k = 1. 

Therefore, according to (88), M = 1. Additionally, the degree of flatness K in this example is 9 

and the filter coefficients are computed using (33). 

Substituting the values of M and N into (90) and (94) we compute different group delays of 

the allpole filter D(z). In particular, we denote the group delay of D(z) based on equations 

(85) and that based on (86) as Ti, and on (91) and (92) as T2. For the design based on (95) 

and (96) we have T3 = T2. The values of Ti, T2 and T3 are —2.875, —2.75 and —2.75 samples, 

respectively. Substituting the values of Ti and T2 and the value of (p^ into (29), we compute the 

corresponding filter coefficients. 

The magnitude responses of the complex wavelet filters based on (85) and (86) are shown in 

Fig. 13a, while the magnitude responses of the complex wavelet filters based on (91) and (92), 

and (95) and (96) are shown in Fig. 14b and 14c, respectively. 

5.5 Fractional Hilbert transformers 

Fractional Hilbert transform has applications in digital communications and signal processing 
(Pei & Yeh, 2000; Tseng & Pei, 2000). There exist different techniques for designing fractional 
Hilbert transformers (FHT) (Pei & Wang, 2002; Tseng & Pei, 2000). Here, we describe a direct 
method for the design of FHT. The method is based on the design of an allpass filter with 
desired characteristic. 
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Fig. 13. Magnitude response of the designed wavelet filters in Example 12. 



The fractional Hilbert transformer is defined as (Pei & Wang, 2002), 



H p (a>) 



_./3/r 

e ] 2 < co < 7i, 



-71 < OV < 0, 






(98) 



where f>, satisfying < ]8 < 1, is the fraction of the Hilbert transformer. 

We can see from (98) that the magnitude response of Ha (to) is 1 and the phase response is 

given by, 

-%£ 0<co<n f 

-71 < CO < 0. 



CHp(w)- ^ ^ 



(99) 



Therefore, the design of FHT is reduced to the design of an allpass filter A(z) with the phase 
response given in (99). If the allpass filter has real coefficients, it is well known that, its phase 
response is an odd function of to, (Mitra, 2005). Consequently, the allpass filter needs to satisfy 
(99) only < to < n. 
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We use a second order allpass filter and w = zr/2 to design the FHT. Therefore, for an stable 
allpass filter, the phase <Pa(w) an d group delay ta(cv) must be — £co — fin/ 2 and £, respec- 
tively, where £ is a positive integer. 
Consequently, from (16) and (18), the design parameters of the corresponding allpole filter are 



(2- 



•fin 



<Pd{ti/2) 
t(tt/2) = 1/2-1. 
To ensure stability, from (25), we have 

£> |sin((* + /3)7r/2)|. 
Substituting (100) and (101) into (23) and (24), we get 

2sin((^ + j 6)7i/2) 



hi- 

/R2 



? + l-cos((^ + ,6)71/2) ' 
?-l + cos((l + ]S)7r/2) 
? + l-cos((^ + i 6)7r/2)' 



(100) 
(101) 

(102) 

(103) 
(104) 



Example 13. The design parameter for the fractional Hilbert transformer are f> = 0.2, 0.4, 0.6, 
0.8, 1 and £ = 2. The resulting phase responses are shown in Fig. 14. 





Phase 


responses 




1 






- 




f> = 0.2 


/ — \ 






_ ._ jg = 0.4 


/ ' \ 




^ 0.5 


£ = 0.6 

jS = 0.8 


//.-■' \ 


v 


3 


/3 = 1 


/• ^ 


V 






/ ^- "** 


^w 


± 








3 


^^ ^v 






§-0-5 


V*- • V 




- 


-1 






- 



0.25 0.5 0.75 

Normalized frequency 

Fig. 14. Designed Fractional Hilbert transformers. 



5.6 New design of MR Butterworth-like filters based on three allpass filters 

We address the magnitude approximation of real-valued lowpass Butterworth-like filters 
based on a new parallel connection of three allpass filters, that is, the proposed structure is 
composed by one real- and two complex-valued allpass filters. The design problem of low- 
pass filter is reduced further to design one complex-valued allpole filter with desired charac- 
teristics. 
The proposed IIR filter is given by 



H(z) 



\ N 



z) + Ai(z)+Ai(z 



-l 



(105) 



The allpass filters Aq(z) and A\ (z) must be stable in order that the filter H(z) be stable. 
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We show that the problem of designing lowpass and stable IIR filter is reduced to designing a 
complex-valued allpole filter with desired characteristics. 
At first, notice that (105) can be rewritten as 

H(z) = ^[l + A(z)+A(z- 1 )], (106) 

where A(z) = Ai(z)/Aq(z) is a complex allpass filter, which can have poles outside the unit 
circle due to the zeros of Aq(z). The complex allpass filter is defined by (14). 
In the following, some characteristics of A{z) are described. 
From (106), the corresponding magnitude response of H(z) is 



\H(e^)\ = ill + e^^+e-^^^l, 



(107) 



where <Pa(w) is the phase response of A(z). 

In order that ^(e^) | has the value 1 in the passband and the value in the stopband (ideal 

case), the condition (pA^) — <Pa( — w) must be satisfied, that is, the phase response must be 

an even function of co. 

Using this property, it follows that A(z) = A(z~ 1 ). Consequently, the magnitude response 

|H(e^)|becomes 

|H(e^)| = i|l + 2cos(^(a;))|. (108) 

Considering the passband edge frequency cop and the attenuation in dB at this frequency point 
Ap. From (108) we define 

i /3.10- A p/ 20 -l\ 
<p vA = cos" 1 I 1 , (109) 

which gives the desired phase <Pa(w) a * <^p, i-e., (ppA — (pAi^p)- 

In order to achieve the condition A(z) = A(z~ 1 ), the corresponding filter coefficients f n , 

n = 0, . . . , N, need to satisfy f n = f-jst-n/ i- e -/ they must be a symmetric sequence. Generally, 

there are two cases that should be considered: N odd and N even. However, one can verify 

that N odd implies that at least one pole of A{z) must be on the unit circle. As a consequence, 

in our design, we only consider the case where N is even. 

Based on (108), We define the following properties for A(z): 

T X We select three frequency points where the phase response <Pa(w) is flat, i.e., co = for 
the passband and co = ±co r for the stopband. Furthermore, <Pa(Q) — an d 0^(±a? r ) = 
2zr/3. This condition ensures that the filter H(z) has flat magnitude response at co = 
and co = co Y . 

T .1 The group delay Ta at co = and co = co r is 0. 

T.3 The phase value (p V A is controlled by A p (see (109)). 

We relate the allpass filter A(z) with the corresponding allpole filter. 

Using (16) and the phase values <Pa(w) at co = and co = ±co r (see Condition T.X), we get 

<p D (0) = and (p D (±co T ) = tt/3 ± co r N/2. 

From (18) and Condition 7.2, we have t(0) = r(zba; r ) = -N/2. 
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Finally, the following relation is obtained using Condition F3 and (16), 

c s-i( ^^ Ua, p N 
foH) = <£ P = ^ * (110) 

As a consequence, the allpole filter D(z) has to satisfy the following conditions: 

Q.\ The phase values of D(z) at co = and a; = ±a;r are and tt/3 ± to Y N /2, respectively 

£.2 The group delay r(o?) of D(z) at a; = and co = ±co r are —N/2. 

Q3 The phase value of D(z) at a? p , cpi){cOp) r is given by (110). 

For a filter having coefficients given in (40) the Conditions C/.l and <?.2 are satisfied. 
From the Condition <?.3 and (41), the corresponding value of cb^ becomes 

tpot(co p ,A v ,to r ) = Z [R p A p f + 1 + jv^(R p + 1)} , (HI) 

where 

P V 1-10" a p /20 
In order to find the value co Y and the order of the allpole filter N, we solve the following set of 
nonlinear equations: 

cpoc (w p , A p , co T ) -cpct(n, A 8 , co r ) = 0, (113) 

<t>a((Vp,Ap,(V T ) -cpcc(to s ,A s ,to T ) = 0. (114) 

Finally, we wish to find the allpass filters Aq (z) and A\ (z) . First note that F(z), the z-transform 
of f n , can be rewritten as F(z) = z~ N ^ 2 F2(z~ 1 )F2(z) / /3, where £2(2) is a polynomial with all 
zeros inside the unit circle, i.e., £2(2) = 1 + /2,l z_1 + • • • + f2,N/2 z ~ N ^ 2 ' an d P — fl,N/2- 
Accordingly, the corresponding allpass filters are expressed as, 



^_ N F 2 (z- 1 )F 2 (z) = N a g F 2 2 (z) 

? 2 (z-l)F 2 (z)' 1U **/3*F|(z) 



_ n; ^^^ a (z)=z -N-iii2^ > (n5) 



Example 14. We design the IIR filter based on three allpass filter using the following specifi- 
cation: co p = 0.371, co s = 0.55n, A p = 0.5 dB, and A s = 45 dB. 

From (111)-(114), it follows that N = 6, co r = 0.64127271, and (pa(tOp, A p/ co r ) = 0.40788971. 
Figure 15 shows the magnitude response of the designed IIR filter. 

6. Conclusions 

In this chapter, we have proposed a new general framework to designing real and complex 
allpole filters with given degree of flatness, and with phase and group delays at any desired 
set of frequency points. The filter coefficients are obtained by solving a set of linear equations. 
In the proposed allpole filter design, we can control the phase, group delay, and degree of 
flatness at different frequency points. Consequently, as demonstrated here, our proposal is 
useful for special IIR filter designs, i.e., linear-phase Butterworth-like filter, Butterworth-like 
filters with improved group delay, complex wavelet filters, fractional Hilbert transformers, 
and new IIR filters based on three allpass filters. 
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Fig. 15. Magnitude response of the designed IIR filter in Example 14. 



Our approach is also useful for the direct design of causal Butterworth filters (Fernandez- 
Vazquez & Jovanovic-Dolecek, 2006) and higher order digital audio equalizers (Fernandez- 
Vazquez et al., 2007). 

As a future work, we will turn our attention to other interesting applications of our proposed 
design. 
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1. Introduction 

Audio diarization Reynolds & Carrasquillo (2005) is the process of partitioning an input au- 
dio stream into homogeneous regions according to their specific audio sources. These sources 
can include audio type (speech, music, background noise, ect.), speaker identity and channel 
characteristics. With the continually increasing number of larges volumes of spoken docu- 
ments including broadcasts, voice mails, meetings and telephone conversations, diarization 
has received a great deal of interest in recent years which significantly impacts performances 
of automatic speech recognition and audio indexing systems. A subtype of audio diarization, 
where the speech segments of the signal are broken into different speakers, is speaker di- 
arization Tranter & Reynolds (2006). It generally answers to the question "Who spoke when?" 
and it is divided in two modules: speaker segmentation and speaker clustering. The goal 
of speaker segmentation is finding the times when there is a change of speaker in the au- 
dio stream. Speaker clustering consists in merging speech segments, detected by the speaker 
segmentation step, related to a same speaker. 

Recently, three main domains of application for speaker segmentation have received special 
attention Reynolds & Carrasquillo (2004): 

- Broadcast news : Radio and TV programs with various kinds of programming, usually 
containing commercial breaks and music, over a single channel. 

- Recorded meetings: meetings or lectures where multiple people interact in the same 
room or over the phone. Normally recordings are made with several microphones. 

- Phone conversations: single channel recordings of phone conversations between two or 

more people. 
Segmenting this types of audio stream in terms of speakers is useful in many application. In 

Automatic Speech Recognition (ASR) Moraru et al. (2003), for example, an initial segmenta- 
tion is required in terms of homogeneous speech and non-speech regions. Having segmented 
speech regions, it is also often necessary to segment these further in terms of homogeneous 
speaker turns. In addition to improving ASR systems, speaker turn information can be help- 
ful for speaker adaptation in rich transcription of videos and meetings Bonastre et al. (2000) 
and for content based audio classification and retrieval Hansen et al. (2005) which have a wide 
range of applications in the entertainment industry, audio archive management, surveillance, 
etc. Audio segmentation would also be an important tool in summarizing meetings, which 
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has recently gained a lot of interest in the research community. For example, segmentation of 
the speech data in terms of speakers could help in efficient navigation through audio docu- 
ments like meeting recordings Dielmann & Renals (2007); Jin & Schultz (2004). Using these 
segmentation queues, an interested user can directly access a particular segment of the speech 
made by a particular speaker. 

1.1 Previous works 

Recent research on audio segmentation mostly focused on four categories: energy based, 
model-based Kemp et al. (2000), metric-based Delacourt & Wellekens (2000), and informa- 
tion criterion-based approaches Cettolo & Vescovi (2003); Chen & Gopalakrishnan (1998); 
M.Cettolo & M.Federico (2000). Energy audio segmentation only detects change-points at si- 
lence segments, which generally are not directly connected with the acoustic changes of the 
audio signals. Model-based segmentation approach requires predefined audio classes and 
complete training data. The metric-based approach are not stable and need thresholds gener- 
ally selected from experiments results. The information criterion-based scheme are proposed 
for evaluating models constructed by various estimation procedures when the specified fam- 
ily of probability distributions does not contain the distribution generating the data. The so- 
called Delta Bayesian information criterion (BIC) segmentation algorithm is widely employed 
in many studies Chen & Gopalakrishnan (1998). The BIC is intended to provide a measure 
of the weight of evidence favoring one model over another. According to previous research, 
the Delta-BIC is threshold-free and suitable for unknown acoustic conditions. However, this 
method, extremely computationally expensive, can introduce an estimation error due to insuf- 
ficient data when the speaker turns are close to each other Huang & Hansen (2004). In order 
to minimize these effects, Delacourt Delacourt & Wellekens (2000)tested different metric cri- 
teria to associate them to the BIC criterion such as the Kullbach-Leibler distance, the similarity 
measure and the Generalized Likelihood Ratio measure (GLR). Still, this method encountered 
problems in case of short segments and requires also a high computation cost. On another 
issue, Zhou Zhou & Hansen (2000)recommends the use of the T 2 — Statistic for metric-based 
segmentation in the aim to reduce this computation cost. However its technique, T 2 — BIC, 
depends on many empiric parameters which affect the quality of the detection of speaker 
turns. In our previous work Kadri et al. (2006), we developed a hybrid segmentation algo- 
rithm called DIS_T 2 _BIC to improve the detection of speaker turns close to each others using 
a fixed threshold independent of the type of the audio stream with a low computation cost. 
Nevertheless all of these techniques suppose that the audio signal don't contains different 
acoustic changes and simultaneous speeches of two or more speakers and then find difficul- 
ties in segmenting streams containing background noise and overlapped speeches. 

1.2 Contributions and Chapter organization 

The main focus of this chapter is to introduce a new unsupervised speaker segmentation tech- 
nique robust to different acoustic conditions. In most commonly used model selection seg- 
mentation techniques like BIC segmentation, the basic problem may be viewed as a two-class 
classification where the object is to determine whether N consecutive audio frames constitute 
a single homogeneous of frames W or two such windows: Wi and W2 with the boundary 
frame or change occurring at the i th frame. In order to detect if a speaker change occurred 
within a window of N frames, two models are built. One which represents the entire window 
by a Gaussian characterized by ]i (mean) , Z (variance) ; a second which represents the win- 
dow up to the i th frame, W\ with \i\iL\ and the remaining part, W2, with a second Gaussian 
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f/2/^2- This representation using a gaussian process is not totally exact when the audio stream 
contains overlapped speeches and very short segments. To solve this problem, our proposed 
segmentation technique use the one class SVM and exponential family model to maximize the 
generalized likelihood ratio with any probability distribution of windows Kadri et al. (August 
2008). Moreover, we use the discrete wavelet coefficient (DWC) to improve the detection of 
speaker changes in the presence of background noise. The use of these coefficient is suitable 
since our technique is insensitive to the dimension of acoustic features. 

The remainder of this chapter is organized as follows. Section 2 details previous audio seg- 
mentation techniques based on BIC. Section 3 reviews the support vector machines approach 
and the exponential family model. The proposed speaker change detection method is illus- 
trated in section 4. Experimental results are provided in Section 5. Section 6 concludes the 
paper with a summary and discussion. 

2. Previous techniques: BIC based segmentation techniques 

Model selection based speaker segmentation is proposed by Chen and Gopalakrishnan Chen 
& Gopalakrishnan (1998). Their method employs the bayesian information criterion as model 
selection criterion, illustrating several desirable properties such as robustness, threshold inde- 
pendence and optimality 

2.1 BIC Segmentation 

BIC Chen & Gopalakrishnan (1998) is a model selection criterion penalized by the model com- 
plexity (amount of free parameters in the model). For a given acoustic segment X;, the BIC 
value of a model M z - indicates how well the model fits the data, and is determined by: 

BIC(X,M) = log L(X lV M,) - £#(M,-) • log(Nj) (1) 

log L(X/, M{) is the log-likelihood of the data given the considered model, Nj is the number of 
frames in the considered segment, #(M Z ) the number of free parameters to estimate in model 
Mi and A is a free design parameter dependent on the data being modelled. A determines the 
'weight 7 applied to model parameters, theoretically 1, but tunable in practice. Given several 
different candidate models to explain a single dataset, the model with the largest BIC gives 
the best fit according to this criterion. 

The BIC-based segmentation procedure is as follows: A sequence of ^-dimensional audio fea- 
ture vectors X = X{ G R d : i = 1, . . . , N are modelled as independent draws from either one 
or two multivariate Gaussian distributions. The null hypothesis is that the entire sequence is 
drawn from a single distribution: 

H = {xi,...,x N } ~ A/^O/^o) 

where N(^,E) denotes a multivariate Gaussian distribution with mean vector \i and full co- 
variance matrix Z. The null hypothesis is compared to the hypothesis of having a segment 
boundary after sample t i.e. that the first t points are drawn from one distribution and that 
the remaining points come from a different distribution: 

Hi :{*!,...,*,} ~ </V(pi,Ei) 

{x m ,...,X N } - A/^2,^2) 
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The difference in BIC scores between these two models is a function of the candidate boundary 
position t: 

where C(X\Hq) is the likelihood of X under hypothesis Hq etc., and (d 2 + 3d) /2 is the number 
of extra parameters in the two-model hypothesis Hi. When ABIC(t) > 0, we place a segment 
boundary at time t , and then begin searching again to the right of this boundary and the search 
window size N is reset. If no candidate boundary t meets this criteria, the search window size 
is increased, and the search across all possible boundaries t is repeated. This continues until 
the end of the signal is reached. 

2.2 T 2 -BIC 

T 2 -BIC Zhou & Hansen (2000) is a variant of BIC segmentation technique which validates each 
speaker change point detected by Hotelling's T 2 -statistic using the BIC criterion. Hotelling's 
T 2 -statistic is a multivariate analogue of the square of the t-distribution Anderson (1985). The 
T 2 -statistic is used when we wish to test if the mean of one normal population is equal to the 
mean of the other where the covariance matrices are assumed equal but unknown. In terms 
of segmentation Wegmann et al. (1999), the problem can be viewed as testing the hypothesis 
Hq : ]i\ = \ii against the alternative Hq : ]i\ ^ \ii where ]i\, ]ii are, respectively, the means 
of two samples of the audio stream, one containing the frame [1, b] and the second contains 
[b, N]. The likelihood ratio test is given by the following T 2 -statistic: 

T2= HN_b 1 ^ i _^ yz _ 1{ ^_^ ) (3) 

where Z represent the common covariance matrix. The T 2 value defined in 3 can be considered 
as a distance measure of two samples. Obviously, the smaller the value of T 2 , the more similar 
the two samples distributions. The T 2 -BIC algorithm operates by fixing an analysis frame 
with L second length from the beginning of the parameterized audio stream and calculating 
the T 2 value in different points situated on this frame; the point that represents the highest 
value of T 2 is more probable to be a real speaker turns; then it can be validated by the BIC 
criterion. The T 2 -BIC segmentation presents certainly some advantages. The selection, from 
the statistical criteria T 2 , of a candidate speaker change permits to reduce computational costs. 
Thus, T 2 -BIC offers a reduced calculation time compared to the BIC segmentation. Besides, 
this technique works with an automatic threshold and presents a low false alarm. However, 
T 2 -BIC is not reliable for the segmentation of audio documents that contain speaker changes 
close to each other. In fact, it requires the use of a time delay r Zhou & Hansen (2000) between 
two consecutive speaker turns which can lead missing some break points. 

2.3 DIS_T 2 _BIC 

Like T 2 -BIC, DIS_T 2 _BIC Kadri et al. (2006) is a speaker segmentation algorithm which pro- 
cess with a fixed threshold and low computation cost. It is proposed to improve speaker turns 
detection even they are close to each other. DIS_T 2 _BIC is based in a hybrid concept which 
is organized in two steps: the detection of most probable speaker turns and the validation of 
changes already detected. Speaker turns are detected by computing the value of T 2 between 
a pair of adjacent windows of the same size shifted by a fixed step along the whole param- 
eterized speech signal. In the end of this procedure we obtain the curve of the variation of 



Robust Unsupervised Speaker Segmentation for Audio Diarization 31 1 

T 2 in time. A speaker change point is characterized by the presence of a high value peak. To 
differentiate high peaks from low peaks, a fixed threshold is defined as below: 

r 2^ (N-2)p c ^ rtA _ T 2 



t > N _ rl V- F -iW 



where Fp /N _ p _i is the F-point for p and N — p — 1 degrees of freedom with significance level oc. 
A T 2 value lower than T 2 shows that the two samples are homogenous and consequently don't 
present a speaker change. So, break points can be detected by searching the local maxima of 
the T 2 curve that verify the criterion 2.3. The validation of already detected break points is 
made using the BIC criterion. Denote {Ti,...,T N } as the set of speaker turns found in the 
first step, a ABIC value is computed for each pair of windows [T;_i, Tj] [T/, T z+1 ]. When this 
value is positive, a speaker turn is identified at time i. Otherwise, the point i is discarded 
from the candidate set, then the ABIC value is applied again for a larger pair of windows 
[Ti_i,Tj + i] [T z+1 ,T z+2 ]. At this stage, when segments are large enough, BIC criterion gives 
better validation results since model estimation becomes more accurate. Detecting speaker 
changes from the curve of T 2 gives to DIS_T 2 _BIC the advantage to detect speaker turns 
close to each others and the use of the T 2 -statistic criteria permits to reduce the computation 
cost and to have an automatic threshold decision independent of the type of the audio stream. 
However, like others BIC based segmentation technique, suppose that the audio signal don't 
contains different acoustic changes and simultaneous speeches of two or more speakers and 
then find difficulties to segment audio streams containing background noise and overlapped 
speeches. 

3. Background information 

This section provides a brief review of reproducing kernel Hilbert spaces, One-class Support 
Vector Machines and exponential families. 

3.1 reproducing kernel Hilbert spaces Aronszajn (1950) 

Let X be a set, and H be a Hilbert space included in the set of all functions on X . The Hilbert 

space % is called reproducing kernel Hilbert space (RKHS) if the evaluation functional e x : 

% 3 f i — > f(x) G IR is continuous on % for any x G X. 

A function k : X x X — > IR is a positive kernel if it is symmetric and for any points x\, ...x n in 

X the matrix (k(xi,Xj))ij is positive semidefinite, i.e., for any sequence of scalar oc\,...oc n the 

inequality Ef /=i &iOLik{x{, xA > is verified. 

Using Riesz's theorem, If H, is a RKHS on X then there exists a function k(.,x) G %, called 

reproducing kernel, such that e x (f) = f(x) = (/(.), /c(.,x))^, where ( , )y > is the inner product 

of H. The function k(x,y) is a positive definite kernel, because it is symmetric from k(y, x) = 



Xj.X 



{k(.,x),k(.,y))<u = {k(.,y),k(.,x))<u = k{x,y), and positive definite from Yj.,] 0L i 0i ^ 

IIEf«ifc(v^)ll«>o. 

In the other hand, it is known that for a positive definite kernel k on X there uniquely exists 
a Hilbert space H^ such that (f(.),k(.,x))<u k = f(x) holds for any / G H^ and x G X. This 
propriety means that 1-L^ is a RKHS with a reproduicing kernel k. given a RKHS % and its 
reproducing kernel k(.,x), because of the uniqueness of the reproducing kernel, we can con- 
clude that the Hilbert space %i constructed by k is identic to %. So there is a bijection between 
the set of all possible RKHS and the set of all positive kernels. 
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3.2 One-Class SVM 

The One-class approach was proposed by Scholkopf Smola & Shawe-Taylor (2000) and has 
been successfully aused for novetly detection. Davy & Godsill (2002) Davy et al. (2006) Des- 
obry et al. (2005). 1-SVM distinguishes one class of data from the rest of the feature space 
given only a positive data set. Based on a strong mathematical foundation, 1-SVM draws a 
nonlinear boundary of the positive data set in the feature space using a parameter to control 
the noise in the training data and another one to control the smoothness of the boundary. 
The 1 -class SVM is a method that aims at learning a single class, by determining its contours. 
To explain 1-class SVM, we can begin by giving a kernel. A kernel k(x,y) is a positive and 
symmetric function of two variables (for more details see [12]) lying in a Reproducing Kernel 
Hilbert Space with the scalar product: 

(f,g)u = LLfi8M x i<yj) ( 4 ) 

In this framework, the 1-class SVM problem with the sample (x z ), i = 1, . . . , m is the solution 
of the following optimisation problem under constraints for / G H : 



^f, P ,d\\f\\u + c ^T=iZi- 


-P 


s.t. f(xi) > p-& i = l,.. 


. ,m 


and & > 0, i = 1, . . . , m 





(5) 



where C is a scalar that adjusts the smoothness of the decision function, p is a scalar called 
bias and £ are slack variables. The dual formulation is: 

(max aG ]Rm ^ oc T Koc 
s.t. oc T e = 1 (6) 

and < oci < C, i = 1, . . . , m 

where K is the kernel matrix K(j = k(xj,Xj) and e = [1, . . .,1] T . The 1-class SVM solution 
is then given by solving a quadratic optimization problem of dimension m under box con- 
straints. The decision function is D(x) = sign(f(x) — p). The input points are considered as 
part of the current class as long as the decision function is positive. 

3.3 Exponential family 

The exponential family covers a large number (and well-known classes) of distributions such 
as Gaussian, Multinomial and poisson. A general representation of a exponential family is 
given by the following probability density function: 

p(x\tj) = h(x) exp{t] T T(x) - A(tj)} (7) 

where h(x) is called the base density which is always > 0, 

tj is the natural parameter, 

T(x) is the sufficient statistic vector 

A(rj) is the cumulant generating function or the log normalizer. 

The choice oiT(x) and h(x) determines the member of the exponential family. Also we know 

that since this is a density function, 

fh(x) exp{7] T T(x) - A(tj)}dx = 1 (8) 
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then, 

A(tj) = log f exp[7] T T(x)]h(x)dx (9) 

For a Gaussian distribution, p(x\}i / a 2 ) = -4= exp(-^x — J^x 1 — £-$ — logcr). In this case, 
h(x) = -4=, t] = [-^2, 2^] and T(x) = [x,x 2 ]. Thus, Gaussian distribution is included in the 
exponential family. 

The density function of a exponential family can be written in the case of presence of an re- 
producing kernel Hilbert space % with a reproducing kernel k as : 

p(x\ti) = h(x) exp{{rj(.),k(x,.))n ~ Ml)} (10) 

with 

i(t]) = \ogjexp{(ti(.),k(x,.)) n h(x)dx (11) 



A( 



4. SVM based speaker segmentation 

4.1 Speaker change detection using 1 -class SVM and exponential family 

Novetly change detection using SVM and exponential family is proposed by Canu and 
Smola Canu & Smola (2005) Smola (2004). Let X = {x\,xi, . . . ,*n} an d Y = {yi,yi, • • • ,yiss} 
two adjacent windows of acoustic feature vectors extracted from the audio signal /where N is 
the number of data points in one window. Let Z denote the union of the contents of the two 
windows having 2N data points. The sequences of random variables X and Y are distributed 
according respectively to Fx and Py distribution. We want to test if there exist a speaker turn 
after the sample x^ between the two windows. The problem can be viewed as testing the 
hypothesis H$ : P x = Py against the alternative HI : Fx ^ Py. Hq is the null hypothesis and 
represents that the entire sequence is drawn from a single distribution, thus there not exist 
a speaker turn. While Hi represents the hypothesis that there is a segment boundary after 
sample X n . The likelihood ratio test of this hypotheses test is the following : 

L(Z1 22N) - — w^m) — - J7 +1 p^) (12) 

since both densities are unknown the generalized likelihood ratio (GLR) has to be used : 

l(zi z 2N )= n ^r4 < i3 > 



i=N+l 



Vx(Zi) 



where P x and Py are the maximum likelihood estimates of the densities. 
Assuming that both densities P x and Py are included in the generalized exponential family, 
thus it exists a reproducing kernel Hilbert space % embedded with the dot product < •, • >% 
with a reproducing kernel k such that: 

P x (z) = h(z) exp{<i/ x (.),t(2,.)>W " M>1x)} (14) 

and 

P y (z) = fe(z) exp{(ti y ak(z,-))H-M%)} (15) 

Using One class SVM and the exponential family, a robust approximation of the maximum 
likelihood estimates of the densities P x and Py can be written as: 
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P x (z) = h(z)exp I ^a[ x) k(z, Zi ) - A(tj x ) ) (16) 



2N 



vWl 



F y (z) = h(z)exp[ £ <^(z,z z -)-A( % ) (17) 

\i=N+l / 

(x) 
where oc\ ' is computed by solving the one class SVM problem on the first half of the data 

(z\ to Z]v), while ocf ' is given by solving the one class SVM problem on the second half of 
the data (z^+i to Z2N")- Using these three hypotheses, the generalized likelihood ratio test is 
approximated as follows: 

J(7 7 , 2 A ^P^N+i^K^d-A^)) 

L{Zi,...,Z 2 N) = 11 ~ 7^ (lo) 

;=N+1 exp (Efll ^ J H Z ]' Z i) ~ Mix)) 

A speaker change in the frame z n exist if : 

2N 2N N 

L( Zl ,...,z m )>s x & E ( E <xr k ( z j' z i)-L*i k ( z i> z i))> s x (19) 

;=N+1 i=N+l z'=l 

where s x is a fixed threshold. Moreover, Y%=n+1 a i H z j' z i) ^ s ver Y small and can be neglect 
in comparison with Y4L1 ocf'k(zpZi). Then a speaker turn is detected when : 

2N N 

£ (-E*,-*(Z;,z,0)>s; (20) 

;=N+1 Z=l 

4.2 Proposed speaker segmentation technique 

In section 4.1, we show that a speaker changes exist if the condition defined by the equa- 
tion (20) is verified. This speaker change detection approach can be interpreted like this: to 
decide if a speaker change exit between the two windows X and Y, we built an SVM using the 
data X as learning data, then Y data is used for testing if the two windows are homogenous 
or not. 

On the other hand, since Hq represent the hypothesis of F x = Py the likelihood ratio test of 
the hypotheses test described in section 4.1 can be written like this: 

L(Z1 22N) = r&V^ = Uw (21) 

Using the same gait, a speaker change has occurred if : 

N 2N 

E(- E *rK*j,*i))><, (22) 

; = 1 i=N+l 

Experimental tests show that in some case is more appropriate when we use Y data for learn- 
ing and X data for testing. Figure 1 presents the segmentation of an audio stream which 
presents four speaker changes. This audio stream is a sample of broadcast news extracted 
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Fig. 1. Segmentation results of an audio stream extracted from NIST RT-02 broadcast news 
data using criteria defined by eq (20) (subplot b), eq (22) (subplot c) and eq (23) (subplot d). 



from NIST RT-02 data. Figures (b) and (c) represent the result of segmentation using respec- 
tively (20) and (22). Using the criteria (20), we can detect only changes number 1 and 3 and 
using the criteria (22), we can detect only changes number 2 and 4. For these reason it is more 
appropriate to use the criterion described as follow: 



2N N N 2N 

£ (-£ a Mfc( Z//Z; )) + £(- £ of Jfcfyz,-)) > S 

j=N+l i=\ ;'=1 i=N+l 

In this case and as illustrated in figure 1, we can detect easily all speaker changes. 



(23) 



4.3 Our segmentation method 

Our technique detects speaker turns by computing the distance detailed in equation (27) be- 
tween a pair of adjacent windows of the same size shifted by a fixed step along the whole 
parameterized speech signal. In the end of this procedure we obtain the curve of the variation 
of the distance in time. The analysis of this curve shows that a speaker change point is char- 
acterized by the presence of a "significant" peak. A peak is regarded as "significant" when it 
presents a high value. So, break points can be detected easily by searching the local maxima 
of the distance curve that presents a value higher than a fixed threshold. 



Algorithm 1: Speaker change detection algorithm 
Step 0: Initialization 

• initialize the interval [a,b],a = 0,b = SIZE_WINDOW 
Step 1 : Computing detection criterion 

• Compute the distance measure dl according to equation (20) with [a,b/2] testing data and [b/2 + \,b\ 
training data. 

• Compute the distance measure dl according to equation (22) with [b/2 + 1, b] testing data and [a, b/2] 
training data 

• Compute the decision criterion d = dl + d2 

• a=a + pas and b = b + pas; go to step 1 
Step 2: speaker turns detection 
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• detecting peaks of d-curve, p = p { 

• decision: 

- if d(pi) > s a speaker change is detected, 

- if d(pi) < s no speaker change is detected, 



5. Experiments 

5.1 Data set 

In order to evaluate 1-SVM-based segmentation method, experiments are based essentially on 
the segmentation of IDIAP meetings Corpus. This database contains two separate test sets 
sampled at 16 kHz. The first test set contains only single speaker segments without overlap- 
ping. However the second one contains a short overlap segment included at each speaker 
change. Further, to generalize our experiments, we used also other types of audio streams 
like broadcast news and telephone conversations. These audio streams are extracted from the 
Rich Transcription-04 MDE Training Data Speech corpus created by Linguistic Data Consor- 
tium (LDC). Description of the used datasets is presented below: 

1. IDIAP meetings Moore (2002): 

• Test set 1: contains only single speaker segments without overlap segments. This 
test set groups nine files, each of them contains 10 speaker turns constructed in a 
random manner with segments duration varying from 5 to 20 seconds. The total 
test set duration was 20 minutes. 

• Test set 2: contains a short overlap segment included at each speaker change. 
The test set is formed by six files, each containing 10 single speaker segments 
(of between 5-17 seconds duration), interleaved with 9 segments of dual-speaker 
overlap (of between 1.5-5 seconds duration). 

2. Broadcast news data: is composed of three approximately 10-minute excerpts from 
three different broadcasts. The broadcasts were selected from programs from NBC, 
CNN and ABC, all collected in 1998. 

3. Telephone conversation: is composed of a 10-minute excerpt from a conversation be- 
tween two switchboard operators. 

5.2 Evaluation criteria 

For evaluating the performance of the segmentation task, we use Type-I errors: precision 
(PRC) and Type-II errors: recall (RCL) was widely used in previous research Ajmera et al. 
(2004). Type-I errors occur if a true change is not spotted (missed alarm) within a certain 
window. Type-II errors occur when a detected change does not correspond to a true change 
in the reference (false alarm). Precision (PRC) and recall (RCL) are defined as below: 

number of correctly found changes 

Total number of changes found 
number of correctly found changes 

Total number of correct changes 

(26) 
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In order to compare the performance of different systems, the F-measure is often used and is 

defined as 

r 2.0 x PRC x RCL 

F = (27) 

PRC + RCL v } 

The F-measure varies from to 1, with a higher F-measure indicating better performance. 

5.3 Audio features components 

In the experiments, two kinds of feature vectors are proposed: MFCCs and DWCs. Mel 
frequency cepstral coefficients (MFCCs) are a short-time spectral decomposition of audio 
that convey the general frequency characteristics important to human hearing. We calcu- 
late MFCCs by using overlapping frames of 30 ms. The Discrete Wavelet Coefficients (DWCs) 
are computed by applying the Discrete Wavelet Transform (DWT) which provides a time- 
frequency representation of the signal. It was developed to overcome the short coming of the 
Short Time Fourier Transform (STFT), which can also be used to analyze non-stationary sig- 
nals. While STFT gives a constant resolution at all frequencies, the Wavelet Transform uses 
multi-resolution technique by which different frequencies are analyzed with different reso- 
lutions. The DWT is computed by successive lowpass and highpass filtering of the discrete 
time-domain signal. This is called the Mallat algorithm or Mallat-tree decomposition Mallat 
(1998). 

5.3.1 Mel frequency cepstral coefficient 

MFCCs are a short-time spectral decomposition of audio that convey the general frequency 
characteristics important to human hearing. While originally developed to decouple vocal ex- 
citation from vocal tract shape for automatic speech recognition. In order to calculate MFCCs, 
the signal is first broken into overlapping frames, each approximately 25ms long, a time scale 
at which the signal is assumed to be stationary. The log-magnitude of the discrete Fourier 
transform of each window is warped to the Mel frequency scale, imitating human frequency 
and amplitude sensitivity. The inverse discrete cosine transform decorrelates these "auditory 
spectra" and the so called "high time" portion of the signal, corresponding to fine spectral 
detail, is discarded, leaving only the general spectral shape 

5.3.2 Discrete Wavelet transform 

The Wavelet Transform provides a time-frequency representation of the signal. It was de- 
veloped to overcome the short coming of the Short Time Fourier Transform (STFT), which 
can also be used to analyze non-stationary signals. While STFT gives a constant resolution 
at all frequencies, the Wavelet Transform uses multi-resolution technique by which differ- 
ent frequencies are analyzed with different resolutions. The DWT is computed by successive 
lowpass and highpass filtering of the discrete time-domain signal. This is called the Mallat 
algorithm or Mallat-tree decomposition Mallat (1998). Its significance is in the manner it con- 
nects the continuous- time mutiresolution to discrete-time filters. In the figure, the signal is 
denoted by the sequence x[n], where n is an integer. The low pass filter is denoted by GO 
while the high pass filter is denoted by HO. At each level, the high pass filter produces detail 
information, d[n], while the low pass filter associated with scaling function produces coarse 
approximations, a[n]. 

5.4 Results 

Table 1 illustrates speaker segmentation experiments conducted on the various audio docu- 
ments previously described and their corresponding results using 1-SVMs and DIS_T 2 _BIC 
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approaches. Segmentation using 1-SVMs outperforms DIS_T 2 _BIC based segmentation tech- 
nique for all the tested audio documents. The segmentation of the IDIAP meetings(l) using 
the two methods presents the highest value of precision and recall. In fact, opposite to other 
types of audio streams, this corpus contains long speech segments allowing good estimation 
of data. As presented in the table 1, the PRC and RCL values obtained with IDIAP meetings(l) 
increases respectively from 0.69 to 0.8 and from 0.68 to 0.79. 



Audio 
Streams 


1-SVM method 


DIS_ 


T 2 _BIC method 




Features 


RCL 


PRC 


F 


Features 


RCL 


PRC 


F 


M. IDIAPl 
M. IDIAP2 

B. News 
Tel. Conv 


39MFCC+DWC 5 
39MFCC+DWC 5 
39MFCC+DWQ 
39MFCC+DWC 3 


0.8 
0.68 
0.75 
0.72 


0.79 
0.67 
0.75 
0.71 


0.79 
0.67 
0.75 
0.71 


13MFCC 
13MFCC 
39MFCC 
13MFCC 


0.69 
0.58 
0.63 
0.56 


0.68 
0.56 
0.66 
0.58 


0.68 
0.57 
0.64 
0.57 



Table 1. Segmentation results using the proposed 1-SVM and DIS_T 2 _BIC methods. 

The proposed method based on 1-SVMs allows the improvement of speaker change detection 
in audio streams which contain overlapping speeches. The improvement in the PRC and 
RCL values using IDIAP meetings(2) is more than 10% with respect to DIS_T 2 _BIC method. 
Generally, BIC based segmentation techniques detect a speaker change between two adjacent 
analysis windows. Each window is modelized by a gaussian distribution. This supposition is 
not true when the window contains overlapped speeches. In this case, it is more suitable to 
suppose that each window can be modelized by an exponential family. 

Broadcast news segmentation results are enhanced by adding discrete wavelet coefficients to 
cepstral coefficients. The use of this kind of parametrization makes speaker changes detection 
possible in the presence of background noise. Further, deploying 1-SVMs permits to better 
put in evidence this characteristic since it is insensitive to the dimension of acoustic features. 
Also, the proposed method is more appropriate to detect speaker changes close each others. 
The F value obtained with the segmentation results of the telephone conversation is raised 
from 0.56 with DIS_T 2 _BIC method to 0.71 with 1-SVMS method. 



6. Conclusion 

In this chapter, we have proposed a new unsupervised detection algorithm based on 1-SVMs. 
This algorithm outperforms model-selection based detection methods. Using the exponen- 
tial family model, we obtain a good estimation of the generalized Likelihood ratio applied on 
the known hypothesis test generally used in change detection tasks. By adding to cepstral 
coefficients the discrete wavelet coefficients. The use of this kind of parametrization permit- 
ted to detect speaker changes even in real-world conditions in which the environment and 
context are so complex that the segmentation results are often affected. The use of support 
vector machines permit to deal practically with this high dimensional acoustic features vec- 
tor. Experimental results present higher precision and recall values than those obtained with 
DIS_T 2 _BIC technique, the increase of PRC and RCL values obtained with various kinds of 
audio streams is roughly over 10%. 
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1. Introduction 

One of the first articles that have addressed lattice quantizers in practical applications is the 
work of Sayood et al. (Sayood et al., 1984). The lattice quantization has been strongly stim- 
ulated by the articles of Conway and Sloane (Conway & Sloane, 1982), (Conway & Sloane, 
1983) presenting fast algorithms for nearest neighbor search algorithms. The principal factors 
that have brought the lattices in the attention of the quantization community are: 

• The lattices are uniformly distributed structures in the n-dimensional space, therefore 
they are immediately applicable as quantizer structures for uniform sources. This af- 
firmation is based on the, now widely accepted, conjecture of Gersho (Gersho, 1979), 
stating that, when the rate is high, the optimal quantizer of a uniform source will have 
the partition cells all congruent to some poly tope. This is equivalent to saying that the 
optimal quantizer of a uniform source is a tessellating 1 quantizer, i.e. it completely fills 
the space with copies of a same poly tope. Gersho (Gersho, 1979) has shown that this 
polytope must have the lowest normalized second order moment for the considered 
space dimension. 

• The asymptotic equipartition property, used in the context of source coding, suggests 
that in a high-dimensional space almost all points will lie in a region of high proba- 
bility specified by the entropy of the source (Cover & Thomas, 1991). The region of 
high probability will have a shape dependent on the source (Fischer, 1989) (e.g. the 
hypersphere for the memory less Gaussian source (Sakrison, 1968), the hyper-pyramid 
(hyper-octahedron) for the memory less Laplacian source (Fischer, 1986)). The pdf of the 
points, /, being almost constant on that region of high probability, the formula under 
the high-rate assumption, for the point density of the optimal quantizer (Gersho, 1979) 
indicates, that the codebook should also be uniform in that region. 

• The regular structure of a lattice VQ reduces considerably the memory requirements for 
the storage of the codebook. 

• Fast nearest neighbor search algorithms exist for the lattices which are generally used 
as quantizers (Conway & Sloane, 1992). 

State of the art speech codecs as AMR-WB+ (Ragot et al., 2004) and G.718 (Ramo et al., 2008) 
codec make use of its advantages. Audio coding methods based on lattice quantization have 
been as well proposed (Vasilache & Toukomaa, 2006). 



1 All the lattices form tessellations, but not all tessellations are obtained from lattices. 
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Most of the lattice based coding methods rely on fixed rate coding or on a semi-variable rate 
coding where the vector to be quantized is split in several sub-blocks for which the rate is vari- 
able, but the overall bit rate for the global vector is fixed (Ragot et al., 2004). There exist also 
variable rate encoding techniques of lattice codevectors. Most of these methods rely on the 
grouping of codevectors on classes such as leader classes or shells (Fischer, 1991), (Vasilache 
& Tabus, 2001) or apply directly entropy coding methods to the lattice codevector components 
(Zhao et al., 2007). However, the former method becomes less practical when the number of 
classes increases (with the increase of the bit rate and for some of the truncation shapes), while 
the latter is from the start less efficient than a direct entropy coding of the lattice vectors in- 
dexes, but obviously less complex. 

We discuss in the present work a new approach for entropy encoding of lattice codevec- 
tors that can be applied for higher dimensional lattices without additional storage require- 
ments and that allows parameterization of the lattice truncation size. The proposed method is 
based on the indexing method for lattice vectors that makes use of the product code indexing 
method. The presented approach is exemplified on rectagular truncation of lattices, where the 
number of leader classes is relatively high, but the shape of the truncation is accounted for 
through companding. 

This work is presenting first several lattice definitions and terms, followed by a short descrip- 
tion of the product code indexing that enables the key method of the work, the new entropy 
encoding of lattice vectors. The proposed method will be exemplified within an audio coding 
scheme that will be briefly presented prior to the results. Future research directions will be 
discussed and conclusions of the work will make the object of the last section. 

2. Lattice quantization: terminology and definitions 

2.1 Lattice definition 

Geometrically, a lattice is an infinite regular array of points which uniformly fills the n- 
dimensional space. 

Algebraically, an n-dimensional lattice A is a set of real vectors whose coordinates are integers 
in a given basis {bj <G R n }i=i /H 

A= |v£R n |v= ^ctibi,cci GZ . (1) 

i=\ 

When used as fixed rate quantizer a lattice should be truncated to a finite number of points 
corresponding to the selected bit rate. Even if, in principle, for the variable bit rate case, 
when entropy coding is applied, the lattice can be considered infinite, for practical reasons 
(i.e. indexing algorithms and numerical aspects of entropy coding), a finite support for the 
lattice should be specified. 

2.2 Lattice truncation 

Generally, the lattice support, or truncation is defined by means of a norm N(x) of the lattice 
points which should be less than a given value K: 

A K ={(x) = (x 1 ,x 2 ,...,x„)eA \N(x)<KJ. (2) 
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The truncation shape is spherical if N is the Euclidean norm, or pyramidal if the N is l\, or 
rectangular if N is the maximum norm i.e. the maximum absolute value of the lattice vector 
components. Also other, more general norms, can be considered. 

A generalization of the rectangular truncation is the truncation having different maximum 
absolute norms, {Ki\i=\ :n along different dimensions 



A Ki = {(x) = (xi,x 2 ,...,x n ) e A \\ Xi \ <K f y 



(3) 



The generalization is exemplified in Fig. 1 for the lattice Z 2 with K\ = 3 and K 2 = 2. The 
truncation includes all Z 2 points inside the smaller rectangle, as well as the points from the 
border. 



Fig. 1. Illustration of the generalized rectangular truncation of Z 2 . 



A given norm defines, in addition to the lattice truncation, the lattice shell, as the set of lattice 
points that have the same norm value, K: 



A K 



{{x) = ( Xl ,x 2 ,...,x n )€A \N(x)=K}. 



(4) 



Consequently, the lattice truncation can be seen as a union of lattice shells. 

A division of the lattice into even finer sets is obtained starting from the definition of a 
leader vector and that of a leader class. A leader vector is a positive integer vector v = 
(v m ,...,v m ,...,Vj,...,Vj,... / Vi,...,Vi) where < V\ < ... < v\ < ... < v m . The leader class of the 
leader vector v is the set of all the vectors obtained through signed permutations, with some 
possible constraints, of the vector v. The leader class notion has been proposed originally in 
(Adoul, 1986), (Adoul & Barth, 1988). 
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Most of the lattices used for quantization can be defined as union of leader classes (Moureaux 
et al, 1998). 

2.3 Counting lattice points 

The use of lattice truncations as quantizers implies knowing the number of lattice codevectors 
inside the considered truncation. Following the definition of a lattice truncation as a union of 
shells, counting the lattice points reduces to finding expressions for the cardinality of a shell, 
i.e. the number of lattice points at a given distance from the origin, under the specified norm. 
The solution of this problem is given by the theta functions for £2 norm for many standard 
lattices in (Conway & Sloane, 1992). In (Sole, 1993) the theta functions have been generalized 
for the norm £p and in (Moureaux et al., 1995), (Barlaud et al., 1993) for weighted £2 norms. In 
(Vasilache et al., 1999) the theta series approach is used to count the lattice points on spherical 
(£2 norm) shells and generalized to pyramidal (£\ norm) shells. 

A second method of counting the points from a truncated lattice is based on the notion of 
leader classes: from the definition of the leader class, the number of vectors belonging to that 
class can be easily deduced using polynomial coefficients (Moureaux et al., 1998), (Rault & 
Guillemot, 2001). This approach is also more appropriate for applications where the indexing 
of lattice points is also required. There exist other methods for counting the lattice points, but 
they will be treated in the section dedicated to the indexing of lattice points. 

2.4 Indexing the vectors in truncated lattices 

Several lattice enumeration techniques have been proposed over the years for different trunca- 
tions and lattice types. One of the first papers to present an indexing algorithm for lattices was 
(Conway & Sloane, 1983), but it was restricted to Voronoi truncated lattices. Few years later, 
Fischer introduced the first enumeration technique on pyramid truncated lattice in (Fischer, 
1986) which he subsequently generalized for weighted pyramids in Z n (Fischer, 1989), (Fis- 
cher & Pan, 1995). This method, which we dub Fischer enumeration, is based on the iterative 
counting 

N(l,k)= £ N(l-l,k-\i\) (5) 

i=-k 

where N(l, k) is the number of vectors in the pyramidal shell of norm k of the lattice Z/. N(l, k) 
can be viewed as the number of ways / integer values can sum up in absolute value to k. For 
maximum efficiency, the numbers N(k,l) must be stored, resulting in a table of size logarith- 
mic in the codebook size. Alternatively, methods of deriving the values of N(l,k) are pre- 
sented in (Hung et al., 1998). A second type of indexing method, also based on an iterative 
counting of the points having a certain property has been presented in (Hung et al., 1998). 
There are four significant quantities of a codevector, which can be iteratively numbered, fi- 
nally their juxtaposition forming a product code. These quantities are: 

1. D(s,l): the number of possible distinct distributions of s elements in / locations, 

2. S(s,k): the number of possible combinations of s non-zero elements that sum up to k 
(distinct additive partitions), 

3. B(s): the number of sign combinations for s non-zero elements. 
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In terms of a lattice codevector, s is the number of non-zero components of the I dimensional 
vector. The number of points for a given t\ norm k is thus given by (Hung et al., 1998) 

m m f]\ fk — 1\ 

N(l,k) = ^B(s)D(s,l)S(s,k) = ^2 s ' r_ (6) 

s=l s=l V / V / 

where m is the maximum number of non-zero elements in the lattice vectors included in trun- 
cation. The use of a product code enhances the error resilience over noisy channels, when 
compared to the original enumeration proposed by Fischer (Hung et al., 1998). These algo- 
rithms, as described in (Hung et al., 1998) apply mainly to Z n lattices or D n with pyramidal 
truncations. The product code of (Hung et al., 1998) can be generalized to spherical trun- 
cations, but with some additional storage requirements for the term S(s,k) (Constantinescu, 
2001). 

In (Serra-Sagrista, 2000) combinatorial formulas like in (6) have been proposed for the A n , D* 
and D+ lattices with pyramidal truncation. A generalization of Fischer's method to lattices 
derived from binary linear block codes through Construction A and B (Conway & Sloane, 1992) 
has been presented for pyramidal truncations in (Wang et al., 1998). This method has 0(nK) 
computational complexity, where n is the lattice dimension and K the truncation maximum 
H\ norm, and it is based on a Fischer type enumeration of pyramidal truncations of Z n and of 
translated 2Z n . 

An indexing technique based on the notion of leader vector of a lattice was developed in 
(Moureaux et al., 1998) for Z n and D n lattices. In (Vasilache & Tabus, 2002) a method based 
on leader vectors for lattices that can be defined as unions of leader classes (including Z n ,D n , 
D* and D+ lattices) has been proposed. Rault and Guillemot (Rault & Guillemot, 2001) have 
presented an enumeration based on signed leaders or generated signed leaders valid for a 
large class of lattices {Z n ,A n , D n and D+ + ). The principle of the methods based on leaders, 
is to count the signed permutations generating the vectors in a leader class. The methods 
described in (Moureaux et al., 1998) and (Rault & Guillemot, 2001) are based on the lexico- 
graphical or inverse lexicographical order of vectors. The methods proposed in (Vasilache & 
Tabus, 2002) utilize also a second possible order of the vectors within a leader class, based on 
binomial coefficients. 

3. Lattice entropy coding 

Allowing variable bit rate encoding through entropy encoding brings substantial compression 
efficiency increase. Moreover, in the case of vector quantization, lattice vector quantization in 
particular, it is more efficient to entropy encode the codevector indexes than the codevector 
components. 

This fact is illustrated in figures 2 and 3 where experimental compression performance in the 
rate-distortion plane is drawn for the lattices D4 and D§ respectively. The curve marked as 
"comp" corresponds to the case when the lattice codevector components are supposed to be 
entropy encoded, while the curve marked with "idx" corresponds to the case when the code- 
vector indexes are supposed to be entropy encoded. The rate is assimilated to the entropy, to 
consider the best achievable case and the entropy values are estimated from the data. Zero 
mean Gaussian data with unitary variance is used for test. Also the curve corresponding to 
the Z4/Z8 lattice is depicted in the graphs, and as expected, for this lattice the rate-distortion 
curves are the same whether the entropy coding is applied to the components or to the in- 
dexes. 
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Fig. 2. Comparison of rate-distortion curves for Z 4 and D 4 lattice when the lattice vector 
components are entropy encoded ("comp") and when the lattice vector indexes are entropy 
encoded ("idx"). 



Ideally, the entropy coding of lattice codevectors should consider each codevector individu- 
ally. However, the use of lattice codebooks is most usefull for high dimensions, where even 
for bit rates relatively small, the number of codevectors easily becomes large, making the indi- 
vidual consideration of each codevector impractical. Practical solutions to this problem have 
been the grouping of codevectors into sets (i.e. shells or leader classes) and entropy encoding 
of the index of the set while the vector index within the set is encoded using enumerative cod- 
ing (Vasilache & Tabus, 2001), (Rault & Guillemot, 2001), (Moureaux et al., 1998), (Loyer et al., 
2003). However, the large number of leader classes for some particular truncation shapes, 
makes their use less practical. 

Another approach has been to entropy encode the lattice vector components (Zhao et al., 2007), 
but for lattices where there exist constraints relative to the values of a lattice vector (e.g. sum 
of components should be even) this approach is not very efficient with respect to the entropy 
coding of the lattice vector indexes. 

3.1 Product code lattice codevector indexing 

In (Hung et al., 1998) the use of a product code type index for pyramidal truncation, in which 
at least the sign bits were separated has been proposed and shown to have good error re- 
silience performance. 

Using a similar approach, the idea of a product code has been extended to spherical lattice 
truncations (Constantinescu, 2001) and to rectangular lattice truncations (Vasilache, 2007). 
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Fig. 3. Comparison of rate-distortion curves for Z§ and D§ lattice when the lattice vector 
components are entropy encoded ("comp") and when the lattice vector indexes are entropy 
encoded ("idx"). 



We propose in the present study the use of the product code indexing from (Vasilache, 2007) 
for the entropy coding of the lattice codevectors. The rectangular truncation uses the maxi- 
mum absolute norm of a vector y = (1/1,1/2, ••-,]) n) £ R" defined as 



N(y) =max(|y i |). 

i=l:n 



(7) 



The idea of the product code is to extract different informational entities from the vector to be 
indexed and concatenate their respective codes. The information contained in the vector from 
a rectangular Z n lattice truncation is represented by the following entities: 

• The number of the significant (non zero) components (A); 

• The number of maximum valued components (in absolute value) (B); 

• The position of the maximum valued components within the lattice codevector (C); 

• The values of the significant non-maximum components (D); 

• The position of the significant non maximum values within the lattice codevector with- 
out the maximum valued components (E); 

• The signs of the significant components (F). 

The borders between the bits corresponding to different entities that form the index are not 
strict, except for the bits corresponding to the signs. The strict border of the sign bits is due 
to the fact that they are situated at an extreme of the index and the cardinality of the set 
describing all the sign combinations is a power of two. The indexing corresponds to the bits 
ordering | A/B/C/D/E | F | . The delimiter " | " represents a strict border. 
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3.1.1 Alternate approaches 

There are more possible representations into information units. For instance, an equivalent 
representation can be (representation II): 

• The number of the significant (non zero) components (A'); 

• The number of maximum valued components (in absolute value) (B'); 

• The position of the significant components within the lattice vector (C); 

• The position of the maximum valued components within the significant ones (D'); 

• The values of the significant non-maximum components (E'); 

• The signs of the significant components (F') 
or as representation III: 

• The number of the significant (non zero) components (A"); 

• The number of maximum valued components (in absolute value) (B"); 

• The position of the maximum valued components within the lattice codevector (C"); 

• The positions of the significant non-maximum values within the lattice codevector with- 
out the maximum components (D"); 

• The values of the significant non-maximum components (E"); 

• The signs of the significant components (F"). 

3.2 Entropy coding based on product code lattice codevector indexing 

The different informational entities extracted from the vector, can be also interpreted as means 
of classifying the vectors into different sets. The existence of several entities implies the divi- 
sion of all the vectors into sets, sub-sets and so forth. If the index corresponding to all or part 
of the set(sub-set) types are entropy encoded, an entropy code can be obtained for the initial 
lattice vector. 

For instance, given the 4 dimensional vector (2 -3 -1), having maximum norm equal to 3, 
it has three significant components (A), one maximum valued component (B), index 1 for 
the position of the maximum valued component (C) and index 1 for the position of the non 
maximum valued components (E) (Vasilache, 2007). There are at least one and at most four 
significant values, therefore there are four possible symbols for the number of significant com- 
ponents, which can be entropy encoded. Furthermore, the number of maximum valued com- 
ponents can be entropy encoded, as well as the position indexes of the maximum valued 
components and so on. 

There is a practical limit to the number of entities that can be entropy encoded, which is 
activated when the number of symbols for the considered entity becomes prohibitively large. 
For instance, for the encoding of the index of non-maximum significative values there are 
(K — 1) S_M possible symbols, where S is the number of significative components and M the 
number of maximum components. For high truncation size (K) and /or high number of non- 
maximum significative values ((S — M)), this number becomes large and the probability of 
the index to be encoded very hard to model. 

For small lattice codevector dimension the proposed lattice codevector entropy encoding 
method might become less efficient than the fixed rate encoding because there are more 
sources of inexact modelling. 
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3.2.1 Lattice truncation size parametrization 

In the previously presented representations of the index, the lattice truncation size, given by 
its norm, is considered to be fixed. A more flexible approach for data with wide range of 
variation is obtained if the value of the current maximum is considered as side information 
that is entropy encoded. 

3.2.2 Context entropy encoding of the index information units 

The encoding of the information units should be done context based, because there is a strong 
correlation between the different units involved. 

For instance, let's consider representation II, when the value of the maximum for each lat- 
tice codevector is transmitted as side information and the variables to be encoded (maximum 
value, number of significant components, number of maxima, position of significant com- 
ponents, position of maxima, index of non-maximum significant values, signs of significant 
components) are denoted respectively by 

K,S,M,pS,pM,nM,sg. 

Then the probability models for each variable are: 

p(K),p(S\K),p(M\S,K),p(pS\S),p(pM\S,M),p(nM\S,M),p(sg). 

For the first variables (K, S, M) their actual values are encoded. For pS and pM a position 
index specifying the location of / components out of n possible locations is created. A position 
vector r = (ro,...,f"/-i) £ 0, ...,n — l l , tq < ... < r\_\ is created, which specifies the exact 
location of each of the I components. Since there are (") such vectors, they can be enumerated 
like binomial coefficients following the algorithm given by the next equations: 

Ipos(Ln,i) = £ f _ 1 + I pos (n - r - 1,1 - 1, (ri,...,rj_i) - r - 1) (8) 

I V os(l f A,\i])=i, 0<i<l' <l. (9) 

The resulting index I pos is the number to be encoded for pS and pM. 

The index to be encoded for the values of non-maximum significant components is calculated 
as 

S-M 
lnM= E^-l) 7 '- 1 ^-!) (10) 

where y z -, i = 0, S — M — 1 are the non-maximum significant values. 

3.2.3 Bit rate calculation 

Consider the n-dimensional vectors from the Z n rectangular truncation of norm K. Any vector 
from this set can be represented on Nq bits, where 

N =riog 2 ((2K + l)")l. (11) 

If the entity corresponding to the number of significant values is entropy encoded on n\ bits, 
the current vector from the set of vectors can be represented on N\ bits instead of Nq, where 
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S is the number of significant components. 

If the number of significant components is entropy encoded on n\ bits, the number of maxi- 
mum valued components is encoded on n^ bits and the index of positions for the maximum 
valued components is encoded on no, bits, then the current vector from the set of vectors can 
be represented on N3 bits, where 

N 3 = Ml+M2 + M 3 + ^l g 2 (2 s Q-^(iC-l) s - M )j (13) 

where M is the number of maximum valued components whose position is already coded on 
ft3 bits. 

If, in addition, the positions of the non-maximum significant values are entropy encoded on 
ft4 bits, then the current vector from the set of vectors can be represented on N4 bits, where 

N 4 = n! + n 2 + n3 + n 4 + flog 2 ((X-l) s - M )J+S. (14) 

4. Lattice quantization for audio coding 

We exemplify the potential of the proposed method within an audio encoding algorithm. For 
the sake of completeness, we present briefly the overall audio encoding framework that uses 
rectangular lattice truncations for quantization. For a detailed description see (Vasilache & 
Toukomaa, 2006). The overall performance of the audio coding method is similar to the 
MPEG4-AAC for higher bitrates (128kbits/s down to 64kbits/s) and better than MPEG4-AAC 
for lower bitrates. 

The global encoding framework is similar to the one used in the A AC. Within the bit pool 
mechanism, at each frame a given number of bits is available for the quantization of the mod- 
ified discrete cosine transform (MDCT) coefficients grouped in several scale factor bands, ac- 
cording to the perceptual model. Roughly, only half of the coefficients are actually quantized, 
the coefficients corresponding to the higher frequencies being set to zero. The number of spec- 
tral coefficients, the number of scale factor bands and their lengths depend upon the sampling 
frequency of the input audio signal. 

The normalized MDCT coefficients from each scale factor band i, are multiplied with b~ Si and 
the result is further encoded. The encoding consists of companding the scaled coefficients 
and quantizing using a rectangular truncation of the lattice Z n . The companding function is 
trained off-line. 

The information to be encoded consists of the scale factor exponents {s z }, the lattice code- 
vector indexes, and side information providing the number of bits on which each index is 
represented. The maximum absolute value, i.e. the maximum norm of the scale factor band 
codevector, is used to calculate the number of bits on which the index of the scale factor band 
codevector is represented. We denote in the following {s z } by scales. 

The scales are integers from a finite domain and they are entropy coded, same as the max- 
imum norms of the lattice codevectors. The scale values are optimized such that the total 
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Name 


Description 


esOl 


Vocal (S. Vega) 


es02 


German male speech 


es03 


English female speech 


scOl 


Trumpet solo and orch. 


sc02 


Classical orch. music 


sc03 


Contemp. pop music 


siOl 


Harpsichord 


si02 


Castanets 


si03 


Pitch pipe 


smOl 


Bagpipes 


sm02 


Glockenspiel 


sm03 


Plucked strings 



Table 1. Test samples. 



number of bits to encode a frame is within the available number of bits given by the bit pool 
mechanism. Since the maximum absolute norm of the lattice codevectors is encoded sepa- 
rately, the indexing of the lattice codevectors is done within the corresponding rectangular 
shell. 



5. Results 

We consider as test samples the 44.1kHz, mono samples presented in Table 1. 
We have considered two encoding bit rates 32kbits/s and 48kbits/s for the audio codec from 
(Vasilache & Toukomaa, 2006). The number of bits for the quantized spectral coefficients is 
calculated according to the formulas from Equations 11 and 12. The difference between the 
average per frame number of bits N\ and Nq for all the spectral scale-factor bands is given 
numerically in percentages in Table 2. It corresponds to the case when the number of signifi- 
cant values is entropy encoded for all the scale-factor bands. The average codelength for n\ is 
estimated based on the entropy. The absolute bit savings are not very significant yet. 
However, when the first three entities (number of significant values, number of maximum 
valued components, and their position index) are entropy encoded, the bit savings become 
significant. 

The difference between the average per frame number of bits N3 and Nq for all the spectral 
scale-factor bands is given numerically in Table 3. The number of bits for the quantized spec- 
tral coefficients are calculated according to the formulas from Equations 11 and 13. 
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File 


BS32[%] 


BS48[%] 


esOl 


6.60 


4.75 


es02 


8.00 


5.62 


es03 


8.80 


6.00 


scOl 


11.20 


7.00 


sc02 


7.20 


5.25 


sc03 


4.80 


3.62 


siOl 


5.40 


2.75 


si02 


9.00 


7.00 


si03 


10.60 


6.75 


smOl 


7.40 


4.00 


sm02 


8.40 


5.25 


sm03 


4.80 


3.25 



Table 2. Bitrate savings, in percentage, when the number of significant values is entropy en- 
coded. 



Confidence intervals 




Fig. 4. Listening test results (Vasilache & Toukomaa, 2006) 



Compared with the number of bits per frame available for spectral quantization only, in the 
fixed rate case, for the considered bitrates, the values in Table 3 give an average of 30% bitrate 
reduction without any loss of quality. 

The method (labeled as 'LatVQ') without entropy coding was compared in (Vasilache & 
Toukomaa, 2006), against the quantization procedure from the MPEG4-AAC codec, in a MUlti 
Stimulus test with Hidden Reference and Anchor (MUSHRA) (BS. 1534-1, 2003). A particular- 
ity of the AAC codec framework was the 11kHz bandwidth considered for quantization for 
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File 


BS32[%] 


BS48[%] 


esOl 


70.20 


53.12 


es02 


37.80 


30.12 


es03 


49.20 


36.62 


scOl 


34.80 


21.75 


sc02 


60.20 


48.75 


sc03 


76.60 


63.87 


siOl 


-21.80 


-6.25 


si02 


-19.80 


-0.75 


si03 


14.00 


9.62 


smOl 


35.40 


28.12 


sm02 


-0.80 


-0.50 


sm03 


46.20 


40.37 



Table 3. Bit savings, in percentages, when the number of significant values, the number of 
maximum valued components, and their position index are entropy encoded. 



all the bitrates. The files used in the tests are listed in Table 1. The files esOl and smOl were 
used only in the training experiment and the remaining files were used in each of the three 
testing experiments. There were 11 expert listeners. 

Since the addition of the proposed entropy coding does not change the quality of the LatVQ 
method, it means that the conditions LatVQ_48 and LatVQ_32 (Figure 4) should actually cor- 
respond to bitrates of approximately 30 % less than 48 kbits /s and 32 kbits /s respectively. 

The proposed entropy encoded method was used in this case only for the scale-factor bands 
with dimensions up to 24, the higher dimensional ones generating too many symbols, at least 
for the position index of the maximum valued components. However, previous entropy cod- 
ing methods of lattice vector indexes were generally on dimension 10 or lower (Vasilache & 
Tabus, 2001). 



5.1 Further discussions 

A very delicate matter related to the enumeration of lattice points is the error resilience over a 
noisy channel. Few papers (Hung et al., 1998), (Vasilache & Tabus, 2002), (Vasilache & Tabus, 
2003) have dealt with the error resilience over the channel for lattice codebooks. 

In (Hung et al., 1998) the channel error resilience is obtained through the use of product code 
based indexing while in (Vasilache & Tabus, 2003) lexicographical and binomial families of 
indexing methods are proposed, allowing the optimization of the indexing with respect to the 
channel distortion (or some other criterion) within a given family. 

The proposed lattice coding, being an entropy encoding method, is on one side sensitive to 
channel errors but is has built-in error concealment mechanisms due to the dependencies ex- 
isting between the information units that are encoded. In addition, extending the observations 
from (Vasilache, 2007), the proposed method can made scalable in bitrate through the control 
of the variables that are encoded, allowing thus an approximate representation of the original 
lattice codevector and the use of the corresponding bits for channel protection, for instance. 
The potential of this approach needs to be investigated through future studies. 
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Another potential direction of investigation is to study the time correlation of each information 
unit to be encoded, that should be easier to exploit than the time correlation of the lattice 
codevector indexes if it exists. 

The method presented here can be applied wherever product code indexing is applicable 
(pyramidal or spherical truncations as well), but it is limited so far to Z n ,D n , D* and D+ 
lattices. 

6. Conclusion 

We have presented a new method for entropy encoding of lattice codevectors. It is based on 
the lattice vector set partitioning generated by the product code indexes of such vectors. It can 
provide bitrate savings up to 30% within an audio coding scenario with respect to the fixed 
rate lattice quantization. In addition to the improved compression efficiency, the proposed 
method enables the use of lattice entropy encoding in higher dimensions. 
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1. Introduction 



Data filtering systems are used in different fields of research, aiming at isolating signals of 
interest from patterns related to a given background noise. Nowadays, in many complex 
applications, the input data space dimensionality is very high, as well as the incoming data 
rate. In this case, the difficulty of the input data stream analysis increases significantly. Also, 
the processing speed plays a critical role when the filtering system is envisaged for online 
operation. Finally, the signals of interest may rarely occur, forcing the experiment to keep 
running for a long period of time in order to acquire a reasonable amount of events for 
better measurement estimation. 
In general, online filtering systems should have the following features: 

• High detection efficiency for a low false alarm probability. 

• Simplified software / hardware implementation. 

• Flexibility in order to accomplish possible future requirements. 

• Execution speed capable of meeting the desired time requirements. 

• Robustness, in order to keep its filtering features through its lifetime of operation. 

To cope with such high-input data dimension, feature extraction techniques may be applied 
in order to isolate the relevant information from the event data description, eventually 
reducing its dimension. For this, different data compaction techniques have been developed 
using expert information or / and stochastic processing. Pre-processing schemes based on 
signal decorrelation (linear or nonlinear) may even reduce the complexity of the classifier 
(signal against background) design. Finally, in the case where the available information is 
from a set of sensors, signal pre-processing might also be segmented, better exploiting the 
available local information. 

Statistical processing can play a valuable role in the pattern recognition task, since it can 
provide better separation cuts than deterministic methods, specially for the case where the 
problem to be solved presents nonlinear characteristics. By using algorithms based on high- 
order statistics, it is possible to better estimate the bounds of each pattern, achieving higher 
detection efficiencies. In many applications, neural networks (Haykin, 2008) may play a role in 
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signal classification. On the other hand, by reducing the classifier design complexity by means 
of signal pre-processing, it might be possible to restrict the nonlinear processing implemented 
by the neural network to perform slight adjustments to the linear signal classification. It may 
also be the case where signal classification can go linear (through a Fisher Discriminant) (Duda 
et al., 2004), as a result of a highly-efficient pre-processing scheme. 

In the field of experimental high-energy physics, stringent conditions make signal 
processing a challenge, as there is often a large gap between the experiment requirements 
and the technology currently available, which forces the development of new technologies. 
This is particularly the case for modern particle collider experiments, in which particles are 
accelerated at high speed and put in collision route. Analyzing the resulting collisions 
products, one can probe deeper into the structure of matter (Perkins, 2000). One important 
aspect in particle collider experiments is that events of interest are typically very rare, since 
most of the produced events are from background noise. In addition, the fine-grained 
segmentation of the particle detectors placed around the collision points for the resulting 
interaction readout may produce up to terabytes per second of information. Therefore, an 
online filtering system must be applied for selecting only the interesting physics channels, 
while rejecting, as much as possible, the huge amount of background noise. 
Presently, the Large Hadron Collider (LHC) at CERN (CERN, 2007) is the largest particle 
accelerator in the world. LHC has a total length of 27 km and will be colliding protons with 
14 TeV at their center of mass, at a rate of 40 MHz and at a luminosity of lO^cm^s -1 (Evans 
and Bryant, 2008). Multiple collision points occur around the LHC ring. Around each 
collision point, a detection laboratory is placed to analyze the sub-products of the collisions. 
Among such detectors ATLAS (The ATLAS Collaboration, 2008) is the largest one. It 
comprises multiple sub-detectors, such as tracking, calorimeter and muon detection 
systems. Due to the detector granularity, each collision produces -1.5 MBytes of 
information, resulting in a total rate of ~60 TB/s of information. Therefore, an online 
filtering system is mandatory for proper ATLAS operation. 

This chapter focuses on proposing an efficient data filtering strategy for operating at 
stringent conditions. It is based on a signal processing scheme that combines expert 
knowledge with stochastic signal processing techniques for data dimension reduction and 
relevant feature extraction. The classifier design that implements the final filtering operation 
(rejection / acceptance of incoming data) is evaluated in terms of complexity and efficiency. 
For this, the input nodes of the classifier are fed from pre-processed information. The 
proposed signal processing strategy will be applied in high-energy physics, using the 
ATLAS detector as a case study. 

This chapter is organized as follows: Section 2 briefly describes the pre-processing methods 
used in the application. Then, in Section 3, the ATLAS filtering system will be explained, 
and the envisaged application is presented. In Section 4, the obtained results for such 
application are discussed. Finally, conclusions are derived in Section 5. 

2. Signal Pre-Processing 

The pre-processing techniques presented in this section focus on performing linear and 
nonlinear input variable decorrelation. This could make the relevant discrimination features 
more evident to the classifier, simplifying its design. Furthermore, depending on the power 
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of the nonlinear decorrelation applied, the classifier could be simplified to the point of a 
simple linear discriminant. 

2.1 Independent Component Analysis 

Independent Component Analysis (ICA) is a multidimensional signal processing technique 

that searches for a linear transformation of the data, so that its essential structure becomes 

somehow more accessible (Hyvarinen et al., 2001). In ICA, the transformed variables are 

restricted to be statistically independent. 

In the standard ICA model, the measured (observed) signals x=[xi,X2,... / xn] t are considered 

to be generated through a linear combination of the independent (unobserved) signals 

s=[si,S2 / ... / s N ] T : 

N 

where z=2,.. v N and A is the mixing matrix. The ICA model has been widely applied in a 
variety of signal processing tasks, see as reference (Choi et al., 2005) and (Moura et al., 2009). 
The purpose of ICA is to estimate the independent signals s and the mixing matrix A using 
only the observed data x. This can be achieved through an inverse model: 

y=Wx, (2) 

where the coefficients of the estimated matrix W are obtained by considering that the 
components of y are statistically independent (or at least as much independent as possible). 
There are some indeterminacies in the ICA model: the order of extraction of the independent 
components can change and scalar multipliers (positive or negative) may modify the 
estimated components. Fortunately these limitations are insignificant in most applications. 
In some practical signal processing problems, the standard linear ICA model may not be 
able to properly describe the data. Considering a practical ICA application, both the mixing 
environment and the sensors may present some nonlinear behavior. Providing a more 
general formulation, the nonlinear independent component analysis (NLICA) model 
considers that the measured signals x are formed by a nonlinear instantaneous mixing 
model (Almeida, 2006): 

x=F(s) (3) 

where ¥(.) is a R N -> R N nonlinear mapping (the number of sources is usually assumed to be 
equal to the number of observed signals). The purpose of NLICA is to estimate an inverse 
transformation G(.) R N -> R N : 

y=G(x) (4) 



so that the components of y are statistically independent. If Gf.^P 1 ^), the sources are 
perfectly recovered (Jutten and Karhunen, 2003). 

A characteristic of the NLICA problem is that the solutions are non-unique (Jutten and 
Karhunen, 2003). If u and v are independent random variables, it is easy to prove that f(u) 
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and g(v), where f(.) and g(.) are differentiable functions, are also independent. So, it is clear 
that, without imposing some restrictions, there is an infinite number of solutions for the 
inverse mapping G in a given application (the problem is ill-posed). Considering this, an 
unique solution for the nonlinear independent component analysis (NLICA) can not be 
achieved without some prior information on the mixing model or the sources. A complete 
investigation on the uniqueness of nonlinear ICA solutions can be found in (Hyvarinen and 
Pajunen, 1999). NLICA algorithms have recently been applied in different problems such as 
speech processing (Rojas et al., 2003), (Wei et al., 2006), image denoising (Haritopoulos et al., 
2002), chemistry sensor array processing (Duarte et al., 2009). 

The minimization of statistical dependence is a main concern for any ICA/ NLICA 
algorithm, as it leads to the estimation of the mixing system (and consequently the 
independent components). In addition, ICA often requires some pre-processing for data 
compaction, especially for high-dimension input data space applications. These topics are 
briefly described in the next subsections. It is also summarized the JADE algorithm, which 
has been widely used for independent component estimation. Different NLICA approaches 
are also briefly reviewed. 

2.1.1 Statistical Independence 

Considering two random vectors vi and v% they are statistically independent if and only if 
(Papoulis and Pillai, 2002): 

/ ? y 1 ,v 2 ( V l' V 2) = ^v 1 (Vi)P V2 (V 2 ) (5) 

where pvi(vi), pv2(v2) are, respectively, the probability density function (pdf) of vi and V2 
and pvi,V2 (vi, V2) is their joint pdf. In typical ICA problems, there is very little information 
on the source signals and so, the pdf estimation is a very difficult task. Considering this, 
alternative independence measures are usually applied during the search for independent 
components (Hyvarinen et al., 2001) (Cichocki and Amari, 2002). They are defined next for 
reference. 



2.1.1.1 Nonlinear Decorrelation 

Two zero-mean random variables (u\ and ui) are said to be (linearly) uncorrelated if their 
cross-correlation R u i U 2 is zero (here, E{.} is the expectation operator): 

K lUl =Z{^\} (6) 

Independent variables are uncorrelated, although, the reciprocal is not always true. Linear 
correlation is verified by second order statistics, while independence needs higher-order 
information too (requiring direct or indirect computation of higher-order moments). 
Considering g(ui) and f(u?) absolutely integrable functions of m and ui, respectively, it can be 
proved that if Equation 7 holds for all possible g(.) and/f.), than m and ui are independent 

E{g(u : )f(u 2 )}= E{g(u : )}E{f(u 2 )} (7) 

By choosing g(.) and/(.) as nonlinear functions, high-order statistical information is (indirectly) 
accessed. The statistical independence measure provided by Equation 7 is usually called the 
nonlinear decorrelation between ui and U2 (Cichocki and Unbehauen, 1996). 
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A practical limitation appears when trying to apply this measure in an ICA algorithm as it is 
not possible to check all integrable functions g(.) and/f.). Thus estimates of the independent 
components are usually obtained while guaranteeing nonlinear decorrelation between a 
finite set of nonlinear functions (Cichocki and Unbehauen, 1996). 

2.1.1.2 Higher-Order Statistics 

Another principle that can be used to estimate the dependence of variables comes from the 
central limit theorem (McClave et aL, 2008): "The sum of two random variables is always 
closer to a Gaussian distribution than the original variable distributions " . As the measured 
signals (x) are considered to be a linear combination of independent sources (s), then the 
measured signals are closer to a Gaussian distribution than the original sources. Thus, the 
independent components can be obtained through maximization of non-gaussianity 
(Hyvarinen et aL, 2001). 

It is known that, for Gaussian random variables, the cumulants of orders higher than two 
are all zero. Considering this, non-gaussianity (and consequently independence) measures 
can be obtained by using high-order cumulants, such as the kurtosis Ka, which, for a zero- 
mean, unit-variance random variable u is defined through (Papoulis and Pillai, 2002): 

K,=E{u 4 }-3[e{u 2 }J (8) 



2.1.1.3 Information Theoretic Measures 

Alternative statistical independence measures can be obtained from information theory 
(Mackay, 2002). A basic definition in information theory is the entropy (H(.)), which, for a 
discrete random variable u, is defined as (Shannon, 1948): 

H(u) = -2>(h = K)\ogP(u = k) (9) 



Where k1, k2, ..., Km are all the possible discrete values of u. is that the Gaussian variable 
has maximum entropy between the random variables of same variance (Hyvarinen et aL, 
2001). Considering this, entropy can be used as gaussianity measure. 
The Negentropy ](u) of the random variable u is also applied in the ICA context:. 

J(u) = H(u gauss )-H(u) (10) 



where u gaU ss is a Gaussian random variable with the same mean and variance of u. The 
advantage of using ](u), instead of H(u), is that it is always non-negative and zero when u is 
Gaussian. A problem with the computation of both ](.) and H(.) is the pdf estimation. To 
avoid this, approximations using high-order cumulants or non-polynomial functions are 
often applied (Murillo-Fuentes et aL, 2004). 

The Mutual Information I(ui, U2, ..., u m ) between m random variables m, Ui , ..., u m is obtained 
through Equation 11. 
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i(u v u 2 ,...,u m )=Ya H ( u )- H (y) ( n ) 

1=1 

It is known that the entropy of the vector v = [ui, U2, ..., u m ] is always smaller than the sum 
of H(m), unless the variables are independent. So, minimization of mutual information 
leads to independence (Hyvarinen et al., 2001). 

2.1.2 Signal Decorrelation 

The standard ICA model assumes a mixing system where the number of sources and 
observed signals is the same. In a practical problem, this assumption may not be always 
true. When there exist more sources than sensors (observed signals), the problem is under- 
determined and the sources are only recovered approximately through algorithms derived 
for such situation (Syskind et al., 2006), (Natora et al., 2009). In the case where the number of 
sources (K) is smaller than the number of observed signals (N), the problem is over- 
determined and thus some signal compaction algorithm is needed to reduce signal 
dimensionality. With this purpose, Principal Component Analysis (PCA) is usually applied 
as a pre-processing for ICA algorithms. Principal Components for Discrimination (PCD) 
analysis (Caloba et al., 1995) has been introduced as an alternative to PCA, when ICA is 
applied to classification problems (Simas Filho et al., 2009b). 

2.1.2.1 Principal Component Analysis 

Principal Component Analysis (PCA) (Jolliffe, 2002) is a statistical signal processing 
technique that searches for a new representation of the input signals where the energy is 
concentrated on a small number of components. Using second-order statistics, PCA 
transformation searches for a vector basis for which the projections yt=Wi Xi of a zero-mean 
random vector x (E{x}=0) are uncorrelated and have maximum variance (i.e. composing an 
orthonormal basis). 
The first principal direction u>i can be computed through the maximization of 

Jl CA {w 1 ) = E{v 2 } = E{(w lX f} = w[C x w x (12) 



where Cx is the covariance matrix of x and | \wi\ | =1. 

PCA transformation is very useful as a pre-processing for ICA as it eliminates second-order 

dependencies (correlation) between the signals, facilitating the search for independence. 

2.1.2.2 Whitening 

A zero-mean random vector z is said to be white if their components are uncorrelated and 
have unit variance (Hyvarinen et al., 2001). This implies that the covariance matrix (and also 
the correlation matrix) of z equals the identity matrix. Whitening is sometimes called 
sphering and is a slightly stronger operation than decorrelation. One popular method to 
perform whitening is the eigenvalue decomposition (EVD) of the covariance matrix (Strang, 
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2009). In this approach, considering Z the matrix whose columns are the unit-norm 
eigenvectors of the covariance matrix C x of a random vector x and D the diagonal matrix of 
the eigenvalues of C x , the linear whitening transform V is given by: 

\ = \y ll2 z T (13) 

2.1.2.3 Principal Components of Discrimination 

Considering a classification problem, the purpose of PCD analysis is to determine the 
directions that maximize class separation (Caloba et al., 1995). Typically, PCD provides a 
higher compaction rate for classification problems with respect to PCA (Simas Filho et al., 
2009b). 

The PCD analysis can be performed through a Multilayer Perceptron (MLP) neural network 
(Hay kin, 2008). For simplicity, considering binary discrimination, a network with a single 
hidden neuron, trained to maximize class discrimination, extracts the first discriminating 
component (see Fig. 1-a). By sequentially adding neurons to the hidden layer and restarting 
the training procedure, the next components are estimated. The hidden weights are trained 
only for the added neurons (highlighted synaptic lines in Fig. 1-b). The estimated weights 
from the previous steps are kept fixed, as they represent the directions of the principal 
components already extracted. The weights of the output layer are adjusted during the 
whole training procedure for optimal combination of principal components at each 
processing step. The PCD extraction continues up to the point where the classification 
efficiency does not improve significantly by adding more components. 

Xi v di 1 - - Frozen 

x^ ^. lm — To be estimated 



X 2 



qi 




d|\|,K UK 

(a) (b) 

Fig. 1. Neural models for estimating (a) the first and (b) the k-th principal discriminating 
component. 

2.1 .3 ICA / NLICA algorithms 

Nowadays, there is a number of efficient ICA algorithms, which use, in general, the 
independence measures mentioned in Section 2.1.1. Among them, JADE (Cardoso and 
Souloumiac, 1993) is a very popular method. For NLICA, one way to address the ill— 
posedness of the problem is to restrict the range of allowed nonlinearities, generating 
structural constrained models for the mixing system and thus unique solutions for the 
problem (Jutten and Karhunen, 2003). Among these models, we can mention the post- 
nonlinear (PNL) mixture, which has met a significant practical applicability (Almeida, 2006). 
There is also a method closely related to the NLICA problem, known as Local ICA, which 
introduces nonlinear transformations by clustering the dataset into groups of similar 
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characteristics. After that, linear ICA is applied to data belonging to each cluster producing 
local independent components. 

2.1.3.1 JADE 

In JADE (Joint Approximate Diagonalization of Eigenmatrices) algorithm, second and 
fourth-order statistics are applied for independent component estimation through a 
tensorial approach. The second-order cumulant (i.e. the covariance matrix) is used to ensure 
that data are white (uncorrelated). Fourth-order information (through the fourth-order 
cumulant tensor matrix) produces an independence criterion. 

Tensors are considered as a higher-dimensional generalization of matrices or linear 
operators (Michal, 2008). Cumulant tensors are matrices containing the cross-cumulants. 
Considering this, the second-order cumulant tensor is the covariance matrix and the fourth- 
order tensor (T4) is formed by the fourth-order cross-cumulants cum(ui,Uj,ujc,ui), which, for 
zero-mean random variables, is defined as: 

cum\u i ,u^,u k ,uA = E^ i ,u i ,u k ,u\-Eiu i ,u\^ (14) 

The fourth-order cumulant tensor T4 is a four-dimensional array, where, for each element 
qijM = cumfuisiipUhUi), the indexes i ,] ,k ,1 vary from 1 to N (where N is the number of signals). 
The fourth-order cumulant tensor contains all fourth-order information of the data. 
JADE estimation criterion is derived through a procedure analogous to diagonalization of 
the covariance matrix, which produces signal decorrelation. As T4 is a fourth-order 
counterpart of the covariance matrix, independence can be achieved by diagonalizing T4, as 
for independent signals the unique non-zero fourth-order cross-cumulant appears when 
i=j=k=l. Analogous to the second-order case, diagonalization of the fourth-order tensor can 
be achieved through eigenvalue decomposition (EVD) (Strang, 2009). 

Using tensorial methods for ICA is theoretically simple, but computing EVD of four- 
dimensional matrices by ordinary algorithms requires a very large amount of memory and 
may be computationally prohibitive in some cases. In order to avoid this limitation, methods 
like JADE were proposed in the literature. JADE algorithm searches for the matrix W that 
minimizes the sum of the squares of the non-diagonal elements of the output data of T/^ 
(where T/^ is the fourth-order cumulant tensor of the output data y). 



2.1.3.2 Post-Nonlinear ICA 
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Fig. 2. Post-Nonlinear mixing/ de-mixing model. 



Demixing system 



Post-Nonlinear (PNL) mixtures arise whenever, after a linear mixing process, the sensors 
present nonlinear behavior. The observed signals can be expressed as (Almeida, 2006): 
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where a=As. As stated in Eq. 15, each observed signal xi is obtained through component- 
wise nonlinear functions fi applied to the linearly mixed signals an. The independent 
components are obtained by a mirror model: 

y, = w,^(x ; .) (i6) 

where W is the de-mixing matrix and gi the inverse nonlinearity (see Fig. 2). The nonlinear 
functions are usually estimated through neural networks (MLP) and the de-mixing matrix 
by a linear ICA algorithm (Taleb and Jutten, 1999). 

A limitation of the PNL algorithm is that the number of observed signals is assumed to be 
equal to the number of sources (square model). This prevents its application to high- 
dimensional data problems as both the number of parameters and the computational 
complexity increase exponentially with problem dimensionality. 

In order to deal with high-dimensional data, a modified PNL model for the overdetermined 
case (when there exist more sensors N than sources K) was proposed in (Simas Filho et al., 
2009a). As illustrated in Fig. 3, a linear block B is added to the standard PLN mixing model, 
allowing K<N. Coefficients of matrix B are estimated through signal compaction methods 
such as PCA and PCD, described in Section 2.1.2. The inverse (demixing) algorithm is thus 
described using a mirror model: y= W G{Dx), where y are the estimated sources. 
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Fig. 3. Modified Post-Nonlinear mixing/ de-mixing model. 



2.1.3.3 Local ICA 

Local ICA (Karhunen et al., 2000), (Jutten and Karhunen, 2003) can be viewed as a 
compromise between linear and nonlinear ICA. If the ICA model is used for feature 
extraction, better description of the data set can be obtained while exploring local 
characteristics. The purpose is to obtain better data representation when compared to linear 
ICA, while avoiding the high computational cost of the nonlinear models. 
In Local ICA model (see Equation 17), a N-dimensional input space Q R N is divided into a 
finite number of subsets Qi, 1=1,. ,.,L, which satisfy: 



Q x yjQ 2 yj...yjQ L =Q 



(17) 



Clustering is responsible for the overall nonlinear representation. Linear ICA models are 
applied to data belonging to each cluster (x(V) in order to estimate the local independent 
components s^=B(W l \ where BQ-) is a local de-mixing matrix. 
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Fig. 4. Local ICA model. 
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3. The Application 

As mentioned in Section 1, the LHC collision rate, together with ATLAS granularity, will 
result in a data stream of -60 TB/s, requiring an efficient online filtering system for 
retrieving the interesting physics channels from dense background noise. This filtering 
system comprises three cascaded operation levels, applying successive cuts to the incoming 
data (RiuetaL, 2008). 

The first level (LI) will receive full data and will reduce the input event rate to -75 kHz. The 
first level is responsible for marking the regions in the detector that have effectively been 
excited. These regions are known as Region of Interest - Rol, and will be the only information 
passed over to the second level analysis. 

The second level (L2) will receive the regions of interest marked by the first level and will 
apply more specific analysis on them. For coping with an average processing time of 40 ms 
per event, a set of 500 off-the-shelf server processors will be employed, providing a multi- 
processed environment. The third and last filtering level, also known as Event Filter - EF, 
will take the final decision on events approved by the previous levels. A highly parallel 
processing environment composed by -1600 off-the-shelf server processors will be 
employed for coping with an average processing time of 4 sec. At the end, a rate of -200 Hz 
events will be recorded in mass storage devices for further offline analysis by the physicists. 
One of the ATLAS main research goals is to experimentally prove the Higgs boson (Perkins, 
2000). Being the Higgs boson highly unstable, it soon decays into more stable particles. 
Therefore, the physicists will prove its existence not by detecting the Higgs boson directly, 
but by analyzing its decaying signatures. It is known (Perkins, 2000) that some of such 
signatures produce electrons at their final state. Therefore, the identification of electrons is 
of great importance. On the other hand, during proton-proton collisions, a cascade of quark 
and anti-quark pairs can be produced, which quickly merge into more stable particles, 
producing a pattern known as jet. These jets may interact with the detector in a manner very 
similar to electrons, making the correct identification of electrons a tricky process. 
Our analysis will focus on the electron / jet separation problem at the second level of the 
ATLAS filtering system, using calorimeter information. Calorimeters are total absorption 
detectors (Wigmans, 2000). Typically, they use a (passive) material (iron, lead, for instance) 
for absorbing entirely the energy of the incoming particle and sample the energy being 
deposited in the detector by using an active material (scintillating fibers, tiles, for instance). 
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Calorimeters play a major role in collider experiments as they provide fast response, their 
energy resolution improves with increasing energy, and they interact with charged and non- 
charged particles. In addition, they are highly-segmented detectors, so that it is possible to 
identify particle classes by their energy deposition profile. 

The ATLAS calorimetry system is composed by two calorimeter sections (The ATLAS 
Collaboration, 2008). The electromagnetic (EM) calorimeter is responsible for detecting 
electrons, positrons and photons. The hadronic calorimeter (HD) is responsible for detecting 
hadrons (kaons, pions, etc) and it is placed on top of the electromagnetic calorimeter. Both 
detectors comprise 3 sequential layers with distinct granularity and depth, providing 
detailed information of incoming particles. The electromagnetic calorimeter has, in addition, 
a very thin layer in front of it, which is called the pre-sampler (PS). Fig. 5 displays the 
energy deposition profile for an electron interaction. A region of interest selected by the 
first-level filtering system amounts, in average, to 1,000 calorimeter cells. 
Electrons have the property of depositing their energy in a very punctual way, differently 
from jets, which, for L2 data, tend to slightly spread their energy over multiple cells within a 
layer. Therefore, the relevant information relies not at the impact point center, but at its 
surrounding area. Aiming at exploiting this feature, a topological pre-processing based on 
ring sums has been tried by some L2 algorithms for data formatting (Torres et al., 2008). In 
this approach, the cell that samples the highest energy value (also known as the hottest cell) 
is considered the center of our region of interest in each calorimeter layer (seven in total). 
Then, a set of concentric rings are built around this hottest cell in a pattern similar to the one 
presented in Fig. 6. It can also be observed in Fig. 6 that, depending on the layer granularity, 
the rings might not close (incomplete) or even be composed only by strips. Finally, the cells 
belonging to a given ring are summed up, reducing the final event dimension, without 
jeopardizing their physics interpretation. This ring procedure is performed on a per layer 
basis, resulting, at the end, in a total of 100 rings, distributed as shown in Tab. 1. 
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Fig. 5. Example of the segmented calorimeter information obtained from an incoming 
electron. 
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Fig. 6. Ring formatting for calorimetry. 



Layer 


PS 


EMI 


EM2 


EM3 


HD1 


HD2 


HD3 


Total 


Rings 


8 


64 


8 


8 


4 


4 


4 


100 



Table 1. Number of rings in each calorimeter layer. 

In the electron / jet separation problem, the dynamic range of the sampled energy is very 
large, therefore, an energy normalization procedure is applied, in order to focus, as much as 
possible, our analysis at the signal shape, rather than its energy nominal value, resulting in a 
steady detection efficiency over all the relevant energy spectrum. Also, since the relevant 
information from the discrimination point of view is known to be off-center, a sequential 
normalization is employed (dos Anjos et al., 2006). In this procedure, for each calorimeter 
layer, the normalized energy (En) of each ring is given by 



-Z£ft> 



(18) 



where En is the original energy of the i-th ring belonging to the 1-th layer, and E to t i is the 
total sampled energy by the 1-th layer. As a result, successively smaller attenuation factors 
are applied to the outer rings, but the normalization procedure is resilient enough to keep 
track of the signal-to-noise ratio, avoiding the amplification of irrelevant information. 



4. Results 

The available dataset was obtained through Monte Carlo simulation for proton-proton 
collisions and comprises approximately 470,000 electrons and 310,000 jet signatures. The 
simulation considers the detector characteristics and the first-level filtering operation. The 
available data set was approximately equally split into training, validation (stopping 
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criterion for neural network training based on mean-squared error) and testing 
(performance evaluation) sets. 




Ring 
Construction 



o 



^( ps ) l=!> | Preprocessing | <s^ 



Neural 
Classification 



y(H3) c£> | Preprocessing | ^ 
Fig. 7. - Processing chain of the electron/jet separation system. 




r~\ (Electron/jet^ 
S/ V decision 



It is shown in Fig. 7 the block diagram of the electron / jet discriminator. The raw 
calorimeter data is received and the topological processing based on ring sums is 
performed. Next, for the segmented case, the rings belonging to a given layer are pre- 
processed individually and the pre-processed event obtained for each layer is concatenated, 
generating a single input, which is propagated to the neural network for the pattern 
recognition. For the non-segmented case, the generated rings are concatenated prior to the 
pre-processing phase, so that the pre-processing is performed in all 100 rings at once. For the 
ICA based pre-processing, the JADE algorithm was used, and the clustering algorithm used 
by local ICA was the k-mean (Duda et al., 2004). 

In this work, the Fisher Linear Discriminant (FLD) and supervised Multi-Layer Perceptron 
(MLP) neural classifiers (single hidden layer) (single hidden layer) were used to perform 
particle identification over calorimeter information. The neural networks were trained using 
the Resilient Backpropagation algorithm (Riedmiller and Braun, 1993). In order to compare the 
discrimination efficiency for the proposed classifiers, both the Receiver Operating 
Characteristics (ROC) and the SP index were applied. The ROC curve (Van Trees, 2003) shows 
how the detection probability Pd and false alarm probability Pf vary as the decision threshold 
changes. The SP index (dos Anjos et al., 2006) is computed through 



SP = ^ L *po*P,) 



(19) 



where Pj is the efficiency for jets. The threshold value that maximizes the SP provides both 
high Pd and low Pf. 

As mentioned in previous sections, the available calorimeter signatures are topologically 
pre-processed, generating 100 rings for an incoming event. Considering this, the 
discrimination system may benefit from signal compaction algorithms as they reduce 
redundant information and signal dimensionality. Here, compaction was performed 
through both Principal Component Analysis (PCA) and Principal Components for 
Discrimination (PCD), using segmented (layer-level) and non-segmented approaches. As the 
calorimeter system provides highly-segmented information, segmented processing tries to 
exploit subtle differences in electron and jet energy deposition profiles, which are available 
at the layer level. 
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Fig. 8. ROC curves (and respective classifier topology) for segmented and non-segmented 
feature extraction through PCA and PCD. 



Fig. 8 illustrates the discrimination performance for different methods in terms of ROC 
curves. It can be observed that the segmented approach outperforms the non-segmented 
one, for both PCA and PCD. It can also be seen that PCD usually presents lower false-alarm 
when compared to PCA (for the same detection probability) and achieves higher 
compaction rates (31 components for PCD against 74 components for PCA in segmented 
processing mode). This is a result of the compaction strategies, as in PCA the purpose is to 
maximize the energy projection and in PCD the objective is to optimize the discrimination 
capability of the components. Moreover, PCD uses nonlinear processing to estimate its 
components, which proves to be efficient in terms of discrimination performance. As it can 
be seen from Fig. 8, using only 31 components, the PCD performance is even better than 
processing 100 rings without any further pre-processing. 

The (linear) Independent Component Analysis (ICA) model was without any further pre- 
processingalso applied to ring signals, either without pre-processing or combined with 
segmented and non-segmented PCA and PCD compaction schemes. Fig. 9 illustrates the 
ROC curves for different ICA-based discriminators. It can be observed that the segmented 
feature extraction provides slightly higher discrimination performance when independent 
components are estimated. Other benefit observed with ICA is that the classifier training 
procedure usually converges in very few iterations, in contrast to PCA and PCD based 
discriminators, which, in general, require a larger number of training steps. From Fig. 9, it is 
also interesting to observe that ICA could be the only pre-processing technique, as the 
nonlinear decorrelation it provides allows the neural network to perform slightly better in 
terms of discrimination efficiency. 

Considering feature extraction through NLICA (using the modified PNL model) based on 
PCD projection, the nonlinearities which may arise are expected to be smooth. In a practical 
design, a calorimeter can exhibit small nonlinearities along the wide dynamic range it has to 
work on. In view of this, the neural networks used to estimate the inverse nonlinearities are 
restricted to have small number of hidden neurons and thus can only approximate smooth 
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nonlinear functions. This also reduces the probability of reaching local minima during the 
training procedure. 
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ROC curves (and respective classifier topologies) for ICA-based discriminators. 



In the Local ICA approach, the training data set was initially clustered into two clusters (as 
there are two possible classes for the incoming particles). As illustrated in Fig. 10, cluster 1 
concentrates most of the electron signatures and cluster 2 the jets. After clustering, ICA and 
ICA with PCD pre-processing were both estimated for data belonging to each cluster. The 
classifiers were also trained locally, generating two ROCs (one for each cluster). A global 
optimization algorithm (Genetic Algorithm) (Haupt and Haupt, 2004) was used to search for 
the optimal combination of the local thresholds, which provides optimum global 
discrimination. 




Cluster 1 



Cluster 2 



Fig. 10. Concentration of electrons and jets in each cluster for Local ICA pre-processing 
approach. 
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Fig. 11 illustrates the discrimination performance obtained through PNL and Local ICA 
approaches. It can be seen that, compared to the linear ICA model, PNL exhibits slightly 
poorer performance. On the other hand, Local ICA produces higher discrimination 
efficiency with respect to the other models when it is performed on PCD directions. For 
Local ICA, only the optimum point is shown in Fig. 11. 
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Fig. 11. ROC curves (and respective classifier topology) for NLICA and ICA discriminators. 

A summary of the results achieved for each approach can be observed in Tab. 2, where the 
maximum SP value obtained for each approach is presented. Furthermore, Tab. 3 presents 
the false alarm probability for a fixed 97% electron detection efficiency. 

The classifier complexity is shown in Tab. 4 for each approach. It can be depicted from this 
table that applying nonlinear decorrelation (PCD and ICA based algorithms) reduces the 
computational requirements for the classification task. The local ICA based on PCD 
projections not only achieves better classification efficiency, but it is also very efficient in 
terms of computational load (~33% reduction with respect to the rings only approach). 



Approach 


Non-segmented 


Segmented 


Rings 


96.10 


— 


PCA 


93.07 


96.04 


PCD 


96.11 


96.28 


ICA 


96.38 


96.45 


ICA + PCA 


93.21 


96.00 


ICA + PCD 


95.45 


96.25 


PNL + PCD 


95.80 


96.20 


Local ICA 


96.63 


— 


Local ICA + PCD 


97.32 


— 



Table 2. Maximum SP (x 100) obtained for segmented and non-segmented approaches. 
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Approach 


Non-segmented 


Segmented 


Rings 


1.75 


— 


PCA 


4.20 


1.88 


PCD 


1.77 


1.59 


ICA 


1.55 


1.51 


ICA + PCA 


4.04 


1.83 


ICA + PCD 


2.26 


1.68 


PNL + PCD 


2.02 


1.72 


Local ICA 


1.30 


— 


Local ICA + PCD 


0.82 


— 



Table 3. False alarm probability (%) for a detection efficiency of 97%. 



Approach 


Non-segmented 


Segmented 


Rings 


3636 


— 


PCA 


4242 


5050 


PCD 


3636 


2828 


ICA 


6868 


8686 


ICA + PCA 


5454 


9090 


ICA + PCD 


200 


3030 


PNL + PCD 


3934 


6768 


Local ICA 


4438 


— 


Local ICA + PCD 


2418 


— 



Table 4. Number of total floating point operations per approach. 



In order to verify whether a linear classifier suffices, the pre-processed signals were used to 
feed a linear Fisher Discriminant (FLD), which is proved to be optimal linear discriminators 
(Duda et al., 2004). Fig. 12 provides a comparison between the discrimination performance 
obtained through linear (FLD) and nonlinear (MLP) classifiers. It can be seen that the 
nonlinear decorrelation introduced by ICA was able to improve the discrimination obtained 
through FLD, providing more separated patterns for different types of particles. Through 
the proposed pre-processing chain, the results of the linear classifier got closer to the ones 
obtained by the MLP. However, the nonlinear decorrelation provided by PCD and ICA were 
still not sufficient to discard a nonlinear classifier in order to achieve optimal detection 
efficiency. Tab. 5 and Tab. 6 summarize the detection efficiency comparison between linear 
Fisher and neural discriminants. An important issue is the computational cost, which, for a 
linear classifier, is much smaller (see Tab. 7) with respect to the nonlinear counterpart. This 
might be a striking advantage for online filtering. 
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0.05 0.06 



Approach 


Neural 


Fisher 


Rings 


96.10 


95.12 


Segm. PCD 


96.28 


95.35 


Segm. ICA + PCD 


96.25 


95.43 


Local ICA + PCD 


97.32 


94.40 



Table 5. Maximum SP (x 100) obtained for each approach considered for a linear classifier. 



Approach 


Neural 


Fisher 


Rings 


1.75 


2.50 


Segm. PCD 


1.59 


2.29 


Segm. ICA + PCD 


1.68 


2.22 


Local ICA + PCD 


0.82 


3.00 



Table 6. False alarm (%) for a detection efficiency of 97% for each approach considered for a 
linear classifier. 



Approach 


Neural 


Fisher 


Rings 


3636 


200 


Segm. PCD 


2828 


200 


Segm. ICA + PCD 


3030 


200 


Local ICA + PCD 


2418 


700 



Table 7. Number of total floating point operations per approach considered for a linear 
classifier. 
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5. Conclusions and Perspectives 

Online data filtering in high-dimensional input data space finds application in multiple 
areas. Depending on the accumulated knowledge about the target problem, combining what 
is known by experts with high-order stochastic signal processing techniques is being shown 
to be an efficient design approach. Among the benefits, high signal compaction rates, 
relevant feature extraction and reduced computational load are often accomplished. 
For a very demanding high-energy physics application, it was shown that we could benefit 
from topological pre-processing, which implements the expert part of the whole pre- 
processing scheme. Then, adding decorrelation techniques to the signal processing chain 
provided efficient feature extraction. For this, only 30% of the original data components 
were required. Further knowledge about the problem pointed out that the segmented signal 
processing was the right approach. In addition, the overall computational load could 
significantly be reduced, which was attractive due to the low processing time required by 
the target application. 

Nonlinear independent component analysis achieved the best performance in this case 
study, which motivates further application investigations. A possibility is to implement it 
through SOM (Self-Organizing Maps) (Haykin, 2008). In detectors where the arising 
nonlinearities of practical designs are expected to be small deviations from the linear 
behavior, it would also be important to restrict the degrees of freedom of the nonlinear 
component extraction. The independent component analysis is also attractive in facing pile- 
up effects (Knoll, 1989), which typically decreases discrimination efficiencies in high event 
rate applications. There is plenty of room for algorithm development in high demanding 
application scenarios. 
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1. Introduction 

The problem of coding the realizations of a random source when some other one, correlated 
with the former, is available at the decoder but not at the encoder goes under the name of source 
coding with side information. The minimum achievable transmission rates in this scenario were 
already found about thirty years ago by means of a random coding analysis. Practical coding 
schemes have been instead investigated only recently for enabling improved compression 
performance in sensor networks and computationally light and robust source coding in video 
applications. 

Differently from the traditional source coding scenario, these schemes take advantage of both 
a code that is good for channel coding and a code that is good for source coding. Practical ap- 
proaches where these two codes are nested have been shown to be asymptotically optimal, 
but schemes that use independent codes have also appeared that are easier to implement, in 
particular in the dual context of channel coding with side information at the encoder. 
In the first half of this chapter we will review the main theoretical results regarding both 
this problem and, in general, the problem of distributed source coding (Section 2). The most 
important coding schemes that have appeared in literature for achieving the promises of the 
theoretical investigations are also described (Section 3); in particular, we discuss an approach 
based on the principle of superposition coding that we also show to be optimal. 
The second half of this chapter is more focused on practical coding schemes. In Section 4 
we give an original factor graph-based interpretation of the decoding algorithms used in the 
schemes for lossless reconstruction based on turbo codes. We also present a performance com- 
parison between several of them. In Section 5 we discuss a solution to the lossy source 
coding problem with side information based on continuous-valued syndromes. In particular, 
this scheme embodies the superposition approach and uses independent channel and source 
codes. In order to broad the range of applications of this coding scheme, model-aided statisti- 
cal decoding of continuous-valued syndromes is also proposed for the case of coding Markov 
sources. We compare the performance of this coding scheme against other systems both for 
the case of coding purely Gaussian sources and for the case of coding natural video sequences, 
both in the discrete cosine transform and in the discrete wavelet transform domain. We will con- 
clude with a short discussion on the drawbacks of the proposed coding solutions and on the 
future research (Section 6). 

Throughout the chapter, we use the following notation. The random variable (r.v) X takes 
realizations x on the set X and has probability mass function p(x). X n is an n-dimensional 
random process with independent and identically distributed components; the realizations x n 
are elements of X n . Matrices and random vectors are shown in bold face (e.g. X). Alphabets 
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are usually discrete. P[] and £[•] denote the probability of an event and the expected value 
of a r.v, respectively; #{•} is the indicator function of an event. H(-), H(- 1 •) and !(•;•) denote 

entropy, relative entropy, and mutual information; A e is the (strongly) typical set over which 
n-dimensional processes distribute uniformly (Cover & Thomas, 2006). A variable X ~ &(jp) 
is a Bernoulli r.v. that equals one with probability p; H(p) is its entropy; addition of Bernoulli 
variables is defined over the group GF(2). A variable X ~ J\f(}i,(T 2 ) is a Gaussian r.v. with 
mean ]i and variance a 1 , a o b denotes function composition, - T matrix transposition; the 
notation 2 nR usually means \2 nR \ . A good source /channel code is a code that achieves the 
rate-distortion /channel capacity function asymptotically with its length. 

2. Problem Statement and Theoretical Results 

Suppose that we want to map some environmental parameter, e.g. the temperature, over a 
certain space. Then, we place m > 2 temperature sensors across that space, and have each 
one communicate its measurement to a central unit; let X z - denote this measurement. Data 
compression is employed at each sensor node in order to save transmission power. It is likely 
that these measurements are somewhat correlated, i.e. that H(XiX2 . . . X m ) < Y4L1 H (^Q). 
According to classical information theory results (Shannon, 1948), at least H(XiX2 . . . X w ) 
bits must be received by the central node for describing exactly all measurements. But this is 
achievable with traditional source coding means only if communication between the sensor 
nodes is possible. The problem of distributed source coding (DSC) refers to the scenario in which 
the sensor nodes are not allowed to communicate with each other. 

Closely related to DSC is the problem of source coding with side information (at the decoder) (SCSI). 
Suppose that the m — 1 measurements X2, X3, . . . , X m are reconstructed at the central node 
upon receiving data from the respective m — 1 sensor nodes. Again, the remaining sensor 
node is not allowed to communicate with any of them. Then, the decoder has knowledge 
about the side information (SI) Y = (X2,X^,. . . ,X m ), which is correlated with measurement 
Xi, but unavailable at the respective sensor node. 

Answers to questions regarding the minimum transmission rates needed in the DSC /SCSI 
problem for lossless or lossy (i.e. within a certain distortion) reconstruction have been mostly 
given and are summarized in the following. Coding schemes achieving these rates have been 
also inherently suggested while answering these questions, but they are in practice not useful. 
Luckily, more structured coding schemes have been investigated in literature that achieve the 
same performance, as described in Section 3. 

2.1 Distributed Source Coding 

Let X denote the source vector (X lr X2, . . . , X m ). A code of length n and rate (R\ f R2, . . . , R m ) 
for the DSC problem consists of the following functions: 

fi : Aff -► [1,2"*'] , z = l,2,...,m, (1) 

m r .. m_ 

g : nM 4 ^"' (2) 

i=\ z=l 

Its probability of error, once p(x) is known, is defined as 

P^ n) = P[S°(flj2 / m )(X")^X"], (3) 
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Fig. 1. Slepian- Wolf coding. 
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and the rate (Ri, R2, . . . , R m ) is said to be achievable if there exist a sequence of codes at rate 

(Ri, R2, • • • ,Rm) such that P^ -^ as n -> 00. The achievable rate region is the closure of the 
set of achievable rates. 

Consider the DSC problem with two encoders (m = 2) shown in Fig. 1(a). The achievable rate 
region for this problem (Slepian & Wolf, 1973) is given by 



R 1 > H(X 1 \X 2 ) / 

Ri > H(X 2 |Xi), 

R 1 + R 2 > H(X 1 X 2 ) , 



(4) 

(5) 
(6) 



and is shown in Fig. 1(b). Each internal point (Ri, R 2 ) of this region is shown to be achievable. 
In particular it is shown that, among the random partitionings of the elements of X- 1 into 2 nRi 
bins each, there exist at least one such that if 

• fi(xf) reveal the bin to which xf belongs, and 

• g(ji,J2) returns the tuple 1 (x\, x^) G Ag with xf belonging to the jf-th bin of Xf , 

the code defined by these functions has asymptotically a negligible probability of error. 
If R\ > H(Xi\X 2 ) and R 2 > H(X2|Xi), a coding scheme is indeed given for achieving a sum 
rate arbitrarily close to H(XiX2), i.e. to the achievable rate when the two encoders can com- 
municate with each other. However, this scheme is not practical because (i) no constructive 
procedures are given to find the needed code, and (ii) there is no structure to be exploited in 
order to evaluate the functions fj and g without resorting to huge lookup tables. Similar re- 
sults can be shown for jointly ergodic sources and for the case with m > 2 (Cover & Thomas, 
2006). 

2.2 Source Coding with Side Information 

Assume that in Fig. 1(a) we have R 2 > H(X 2 ). Then, by classical results, the decoder can 
already reconstruct X2 without receiving any data from the first encoder. We may then wonder 
which is the minimum rate needed for reconstructing X = Xi when the SI Y = X 2 is available 
at the decoder, as in Fig. 2(a). From the analysis of the DSC problem, it can be inferred that 
lossless reconstruction of X is possible at rates over H(X\Y). 



1 More precisely, if there are no tuples or there is more than one that satisfies this property, g assigns a 
random tuple to (j\,J2). Similar strategies are also taken by the " theoretical" algorithms discussed in 
the following. 
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Fig. 2. Wyner-Ziv coding. 




(b) virtual channel 



From a broader point of view, it is also interesting to investigate the rates needed for lossy 
reconstruction with SI. If X is the set over which the reconstruction X of the source at the 
decoder takes values, a (single-letter) distortion function d : X x X —> IR + is usually defined that 
is extended to n-dimensional realizations by assuming (with an abuse of notation) d(x n , x n ) = 

In the SCSI scenario, a code of length n and rate R consists of the following functions 



/:#"-> [l,2" R 
g : [l,2 nR ] x y n ^ X n . 
Its distortion, once p(x,y) is known, is defined as 

D(") = E[rf(X",^(/(X"),Y M ))] , 



(7) 
(8) 

(9) 



and the pair (R,D) is said to be achievable if there exist a sequence of codes at rate R such 

that lim n ^oo D^ < D. The rate-distortion region is the closure of the set of these achievable 

pairs; the rate-distortion function R*(D) is the infimum of rates R such that the pair (R,D) is 

achievable. 

The rate-distortion function for SCSI equals (Wyner & Ziv, 1976) 



R*(D) 



min I(X;li|Y), 

pO|x),p(%,y) 



(10) 



where the minimum 2 is taken over all variables U and X such that Y — > X — ► U and 
X — >- (LT,y) — >- X form a Markov chain, and E[d(X,X)] < D. It is interesting to note 
that the rate-distortion function when both the encoder and the decoder have access to 
the SI, i.e. R x \y(D) = min/(X;X|Y) where the minimum is over all p(x\x,y) such that 
E[d(X, X)] < D (Shannon, 1959), can be similarly expressed as 



Rx\y(D)= min I(X;X|Y) 

p(u\x),p(x\u,y) 



(11) 



where the minimum is taken under the same conditions of equation (10) (Pradhan et al., 2003). 
Since I(X;U\Y) > I(X;X|Y), in general, R*(D) > R X \ Y (D), with equality if and only if at 
optimality in(ll)X— ^ (X, Y) —¥ U forms a Markov chain. 

The rate loss with respect to the case where also the encoder knows the side information 
is investigated in (Zamir, 1996). However, the Gaussian-quadratic scenario is an important 



■ The search can be limited to p(x\u,y) — x{% — h(u,y)}, where h is a deterministic function. 
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example in which R*(D) = R X \ Y (D). In particular, if X — X — IR and d[x, x) = (x — x) 2 , 
there are no losses if (X, Y) ~ Af (Wyner & Ziv, 1976) and also if X = Y + N, with N ~ J\f 
independent from Y, which in turn can have any distribution (Pradhan et al., 2003). 
In SCSI, the coding scheme used in the proof of achievability of all rates over R* (D) involves 
the random generation of a set U of 2 nl ( x ' /U > codewords distributed according to Ylf=\ P*( u i)s 
where p*(u) is the marginal of the joint distribution achieving the minimum in (10), and the 
random partitioning of them into 2 nI ( X;LI l y ) bins; each bin contains approximately 2 nI ( Y;LI ) ele- 
ments. The encoder associates each x n to the element u n <G U such that (x n ,u n ) <G J\f , and 
sends the index j of the bin to which u n belongs. The decoder looks for the element u n in the 

(n) 
j-th bin such that (u n ,y n ) <G A\ , and returns Xj = h* (u^yi), where h* (u,y) is the determin- 
istic function achieving the minimum in (10). Note that the bins into which U is partitioned 
take the role of good channel codes for the channel between Y and XI; in turn, U takes the role 
of a good source code, since via the function h it leads to a representation close to the source. 
Again, this scheme is practically useless. 

In many cases the decoder has perfect knowledge about the SI. One may however wonder if 
it is possible to characterize a rate-distortion region assuming that both Xi = X and X2 = Y 
are independently encoded at finite rates Ri and R2 and jointly decoded into representations 
within distortions D\ and D2, respectively. In the general case, this rate region is currently 
still unknown and only inner and outer bounds have been found. Recently, the problem has 
been solved for the two-terminal Gaussian-quadratic case. We refer the interested reader to 
(Wagner et al., 2008) and to the references therein. 

3. Structured Codes for Binning 

In practice, in order to achieve the performance claimed by the theory there must exist good 
structured codes and feasible algorithms to search for jointly typical codewords over them. Trans- 
mission rates involved in the DSC /SCSI problem are also finite. Lossless reconstruction is 
therefore possible only when dealing with discrete source alphabets. All practical lossless 
coding schemes treat essentially the binary case, that is the most simple; on the other hand, 
the binary representation of a discrete r.v can be always seen as a random vector with (corre- 
lated) binary entries. Lossy reconstruction is instead possible with both binary and continuous 
(real) variables; the Hamming distance 

<»(*>* -{I: TA (i2) 

and the quadratic distance d(x,x) = (x — x) 2 are normally used as distortion function in the 
two cases, respectively. In the following we will review some of the practical coding schemes 
recently appeared in the literature for the DSC /SCSI problem. 

3.1 Near Lossless Coding 

In traditional binary lossless source coding there exist many exactly lossless codes with perfor- 
mance close to the Shannon bound. On the other side, practical codes for binary channel coding 
with rates close to the Shannon capacity bound, such as the turbo codes (Berrou & Glavieux, 
1996), operate in a near lossless regime, i.e. with very low, but not zero, probability of error. 
As observed in (Wyner, 1974), the problem of lossless SCSI is indeed a channel coding prob- 
lem. Assume that the source and the SI are actually generated from two independent binary 
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sources Y and Z ~ Z3(p) as in Fig. 2(b); then, a virtual "correlation" channel exists between Y 
and X. This channel is a (memoryless) binary symmetric channel (BSC) for which linear codes and 
closest neighbor search algorithms are asymptotically shown to achieve capacity 
Let H be a parity-check matrix of a good [n, k) code C for the virtual channel. The code contains 
all binary vectors c n whose syndrome s n ~ k = Hc n is equal to zero, and partitions X n into 
2*i -k cose f S (bins) of 2 k elements; elements in each coset C s have a distinctive syndrome s n ~ k . 
The code C can be used to reliably transmit information at its code rate Rq — k/n « C bit 
per channel use, where C = 1 — H(Z) is the channel capacity (Cover & Thomas, 2006). The 
decoding function is defined by fc (s n ~ k ) = arg min c n eCs dn (c n , 0), and is such that P[f(HZ n ) ^ 
Z n ] is asymptotically negligible. Then, the code defined by (see equations (7) and (8)) 

f(x n ) = Hx n , (13) 

g{s n - k ,y n ) = y n +f c (s n - k + Hy n ) (14) 

is a good near lossless code for the SCSI problem. In fact, its rate is R = (n — k)/n = l — Rc~ 
H(Z) = H(X|Y),and 

P[g(f(X n ),y n ) ^ X n \ = P[Y n +f c (HZ n ) ^ X n ] (15) 

is asymptotically negligible. 

Practically, the capacity of the BSC has been approached by turbo and low density parity check 
(LDPC) codes (MacKay, 1999). These codes have been successfully used in SCSI too. The 
evaluation of the syndrome in order to perform encoding is straightforward in the case of 
LDPC codes and very easy in the case of turbo codes. SCSI approaches based on turbo and 
LDPC codes are investigated in, among others, (Liveris et al., 2003; Roumy et al., 2007; Tu 
et al., 2005) and in (Liveris et al., 2002), respectively. In all cases the decoding function is very 
well approximated by means of iterative algorithms; we will give some more detail on these 
algorithms in Section 4. 

With reference to Fig. 2(b), if Y ~ 8(1/2), then it turns out that Z is independent from X too, 
such that a BSC is indeed also defined by Y = X + Z. SCSI approaches based on systematic 
turbo codes use this interpretation to show that if the n — k parity bits relative to each encoded 
fc-tuple are sent to the decoder, then near lossless performance is achieved since the SI is indeed 
a corrupted version of the systematic portion of a codeword (Aaron & Girod, 2002; Garcia- 
Frias & Zhao, 2001). 

Under the same condition, a coding scheme based on syndromes was also devised for the 
two-terminal DSC problem that achieves all rates with the minimum sum rate. Let G and H 
be a generator matrix and a parity-check matrix of a good (n, k) code C for the virtual channel 
between X2 = Y and Xi = X. Chose two subcodes C\ and C2 of C that admit generator 
matrices Gi and G2 constructed by taking k\ < k and the remaining ki = k — k\ rows of G, 
respectively. Each c n e C has a unique factorization into the sum of two codewords belonging 
to these two subcodes; denote with 7i z - the function that gives the codeword relative to code 
Cj. Denote with H z - a parity-check matrix for the code C\, and with tj(s?~ ') a deterministic 
function that gives an n-tuple of C Zs . . Define the function 

Ms^ 1 ^^)^ (f 1 (s^) + f 2 (s 2 "-^))+/ c (H(f 1 (sr^) + f 2 (s^fe))) . (16) 
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Then, the code defined by (see equations (1) and (2)) 

/,(*?) = H iX f, (17) 

has rate that satisfies R,- > (n - k)/n « H(X a |X 2 ) = H(X 2 |X 1 ) and R 1 +R 2 = {In -k)/n = 
2 - R c w 1 + H(Z) = H(X X X 2 ), and 

Pfe(/l(X 1 "),/ 2 (X 2 "))/Xf] (19) 

is asymptotically negligible. This is a direct consequence of the fact that, if c" G C/ is such 
that ^( H **f) = x" + cf, P[/z(H 1 X^,H 2 X^) ^ C[ + C£] is asymptotically negligible since 
X 1 + X 2 ~B(p). 

The scheme outlined above for encoding two sources was devised in (Pradhan & Ramchan- 
dran, 2005) and essentially tested using convolutional codes. Convolutional codes are used in 
(Pradhan & Ramchandran, 2003) too in the SCSI context. The extension to m > 2 sources is 
delineated in (Stankovic et al., 2006), and it is shown to be optimal for uniform sources such 
that Y4L1 Xi ~ &(p)'' tests conducted with more performing codes (e.g. turbo) show a very 
good performance. 

3.2 Lossy Coding 

In case of reconstruction within a certain distortion, the search for practical SCSI algorithms is 
more involved since both a good channel and a good source code are needed. In the seminal 
work (Zamir et al., 2002), it is shown that nested linear codes and lattices achieve the theoretical 
bounds in the binary-Hamming and in the Gaussian-quadratic case, respectively. 
In practice, the suggested coding scheme requires that a fine linear /lattice code C\ is used that 
is a good source code for the problem of coding the source without SI. Instead of sending 
to the decoder the index of the codeword that is the closest to the actual realization of the 
source (as in traditional source coding), the encoder sends the index of the coset to which 
this closest codeword belongs. The codewords of C\ are in fact partitioned into cosets once a 
coarse linear /lattice subcode C2 C C\ is defined. If C2 is a good channel code for the virtual 
channel between the SI and this closest codeword, then the decoder can reconstruct the latter 
with negligible probability of error, as shown in the previous section. From it, and using also 
the SI, an estimate of the source is eventually obtained. The theoretical bounds are achieved 
because, with good linear /lattice codes, (i) the closest codeword distributes as the variable 
U n rsj nf=i P* ( u i)/ (ii) a t the decoder, once u n is reconstructed, the distortion is minimized by 
implementing the function h* (u,y), and (iii) the number of cosets, i.e. the nesting ratio of the 
two codes is approximately I(X; U\Y). 

In lossless SCSI the virtual channel considered for channel code design is between Y and X; 
here, instead, it is between Y and U. But Y —> X — >• U forms a Markov chain, so the channel 
code must be actually more robust; this phenomenon is investigated in detail in (Zamir et al., 
2002) under the name of self-noise. 

In particular, in the binary-Hamming case (Z ~ &{jp)) optimality in (10) is achieved by U = 
X + Q with Q ~ B(D) independent from X, and by h*(u,y) = u (see Fig. 3(a)). In practice, 
dithered quantization over a fine code guaranteeing distortion D achieves almost this U (Zamir 
& Feder, 1996), but with a coarse channel code that is exponentially good for the BSC U = 
Y + Z e , where Z e = Z + Q - B(p * D) with p * D = p(\ - D) + D(l - p), the recovery of LI 
from the SI at the decoder is still successful (Zamir et al., 2002). 
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(a) binary-Hamming case 



(b) Gaussian-quadratic case 



Fig. 3. Relations among the random variables involved in lossy SCSI at optimality: Y, Z 
and Q are independent Gaussian /Bernoulli random variables. U is not actually "sent" to the 
decoder, but rather reconstructed relying on the knowledge of Y. 



In the Gaussian-quadratic case (Z ~ jV(0, cr|)) optimality in (10) is achieved by U = ocX + Q 
with Q ~ AT(0, D) independent from X, and by h* (u,y) =ctu + (l- oc 2 )y; oc = y/l - D/c% < 
1 guarantees that (i) h* (u,y) gives the minimum mean square error estimate of X from Y and 
U, and (ii) E[d(X, h* (U, Y))\ = D (see Fig. 3(b)). In this case, dithered quantization of ocX over 
a fine lattice guaranteeing distortion D achieves essentially this U (Zamir & Feder, 1996) 3 ; if 
the coarse sublattice is exponentially good for the additive white Gaussian noise (AWGN) channel 
U = ocY + Z e , where Z e = ocZ + Q ~ A/"(0, cr|), then U can be recovered from ocY at the decoder 
(Zamir et al., 2002). 

The nested coding approach outlined above, for which the existence of good and nested codes 
was actually showed in later papers, is surely more practical than random binning; practical 
implementations of the Gaussian-quadratic case are indeed investigated in (Liu et al., 2006; 
Servetto, 2007). However, there are still practical difficulties in its implementation due to the 
fact that good performance is achieved only asymptotically with the length n of the code. In 
practice, for finite n recovering of U is possible with the coarse code having a reduced code 
rate with respect to the one achieving optimality, not only because the self-noise (Q) actually 
introduced is not Bernoulli /Gaussian, but also because the nesting constraint does not permit 
to efficiently randomize the channel code. This leads to a rate loss with respect to the achiev- 
able rate, or, equivalently, to a distortion gap, which in particular increases for fixed n with the 
rate of the SCSI scheme. This phenomenon was observed in (Liu et al., 2006), and a scheme 
was proposed in which a second coding stage (based on schemes shown in the previous sec- 
tion) on top of nested quantization with finite n is used to losslessly send the coset index 
to the Si-aware decoder; the resulting scheme is probably the most performing appeared in 
literature. 

Other practical schemes for lossy SCSI of continuous sources are based on the observation 
that once traditional (vector) quantization produces an index in a discrete domain, lossless 
SCSI means could be used to send this index to the decoder. In this way, a good source code 
and a good channel code are indeed individually designed. Channel codes that are essentially 
based on convolutional codes are used in (Pradhan & Ramchandran, 1999; 2003; 2005); more 
performing channel codes are used instead in (Yang et al., 2008), where the problem of lossy 
distributed coding of many sources is tackled as well. As a remark, note that even if this 
"separated" approach has indeed been shown to be optimal for the two-terminal Gaussian- 
quadratic case (Wagner et al., 2008), in general it is not. 



3 Note that, if Y ~ J\f, U' = ocll assumes the conditional distribution p(u'\x) which minimizes I(X;U') 
under distortion D (Cover & Thomas, 2006). 
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3.2.1 Superposition Coding 

It should be clear that the joint design of good source /channel codes for the lossy SCSI /DSC 
problem is very hard. Coding schemes in which optimality is also achieved if the two codes 
are designed independently are then preferable since they are more practical. One such 
scheme has been proposed and investigated for the dual problem of channel coding with side 
information (at the encoder) (CCSI) and is briefly discussed in the following. The SCSI approach 
we propose in Section 5 is indeed dual to this approach; more details regarding this duality 
can be found in (Cappellari, 2009). 

Consider the additive channel X = X -\-Y -\- Z, where X is the input to the channel, Y is 
an interference known to the encoder (but not to the decoder) and independent from X, Z is 
an unknown noise independent from X and Y, and X is the channel output. The goal is to 
transfer the maximum amount of information to the decoder once a cost constraint is given 
to X; usually this constraint is given as E[d(0,X)] < W where d is a distortion function. A 
capacity-cost function C* ( W) is then defined that gives this maximum amount; in particular, it 
was evaluated (for the general, non-additive setting) in (Gel'fand & Pinsker, 1980). 
A random coding scheme is suggested too as follows. Generate a set U of codewords dis- 
tributed according to a suitable distribution Ilf=i P*( w z')/ an d partition it into bins; associate 
each bin to a different message to be sent. The encoder selects the u n in the desired bin such 

(n) 
that (u n ,y n ) G A e and inputs to the channel the codeword f z - = n*(n z -,y z ), for a suitable 

choice of h*. The decoder looks for the element u n in hi such that (x n , u n ) G Ag , and returns 
the message corresponding to the bin to which this element belongs. Now, the bins into which 
U is partitioned take the role of good source codes, since via the function h a codeword satisfying 
the cost constraint can be formed and transmitted; in turn, U takes the role of a good channel 
code. 

In the additive setting, nested linear codes and lattices exist that achieve optimality both in 
case of binary-noise Hamming-cost and in case of Gaussian-noise quadratic-cost, respectively 
(Zamir et al., 2002). Nevertheless, optimality is also achieved with a superposition of two in- 
dependent codes (Bennatan et al., 2006). In particular, a coarse code C^ which is a good source 
code with respect to the distortion constraint W is used by the encoder for selecting the code- 
word to be transmitted on the channel. The fine code C\ is generated as the sum of C2 and a 
code Co which is a good channel code with respect to the noise Z, i.e. C\ = Cq + Ci> With a 
proper generation of Co, C\ is partitioned into bins Cq + Ci, for Cq G Co, i.e. each Cq G Co is 
in one-to-one correspondence with a bin of C\ . At the encoder the message is then selected 
in terms of a Cq G Co- At the decoder a codeword of C\ plus some noise is received; upon 
decoding this signal over the sum of Co and C2, the best estimate of Cq is declared as the re- 
ceived message. Details and implementation of this scheme with practical codes is discussed 
in (Bennatan et al., 2006). 

In the lossy SCSI problem, let us in principle generate a coarse code C2 which is a good channel 
code with respect to the virtual channel between Y and U. Then, we generate Co as a good 
source code for Z e with respect to the distortion D, and obtain the fine code C\ = Co + Ci> In 
particular, in the binary case (Z ~ B(p)) C2 and Co are codes made of 2 nRl and 2 nR ° code- 
words with distribution 8(1/2) and B(q), respectively. The encoder looks for eg + c\ such 

that (x n ,CQ + dj) G Ae and sends the index of Cq (in practice, dithered quantization can be 
employed); there is an encoder error if dn(x n , c[J + dj) > D. 
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H(X)-H(p*D) ■ 

H(X)-H(q*D) - 



Fig. 4. Region in which there are no encoder nor decoder errors. 



Theorem 1. The probability of having an encoder error is negligible with n — » oo if 

R 2 > H(X) - H(q * D) (20) 

Rq + R 2 > H(X)-H(D). (21) 

Sketch of proof: Asymptotically, x n takes values on a set of 2 nH ( x ) elements, and for a fixed dj, 
the code Cq + c^ covers at most 2 nH (^* D ) of them within distortion D. Hence, there must be 
at least 2 nH ( x )~ nH (^* D ) codewords in code C2. Then, since each n-tuple approximates at most 
2ftH(D) e i emen ts within Hamming-distortion D, the superposition code must provide at least 
2nH(x)-nH(D) codewords. Once we have these two conditions the probability that a typical 
(cq + c\ ) is within distortion D approaches one. This may be proved with an argument similar 
to the one used in the standard proof of the achievability of the rate-distortion function (Cover 
& Thomas, 2006). If both given bounds are approached, the test channel between Cq + C2 and X 
is such that X = Cq + C2 — Q, with Q ~ B(D) independent from Q and C2 (asymptotically); 
for rates over the bounds a lower distortion can be achieved. 

(n) 

The decoder looks for c\ £ C 2 such that (eg + c\,y n ) £ *4g 7 (in practice, a closest neighbor 
search is conducted), and reconstructs the source as eg + c\) there is a decoding error if c\ 
is not correctly recovered. But once Cq is known, the equivalent noise between Y and C2 = 
Y + Z + Q — Cq is distributed as Z e . Hence, the following holds. 

Theorem 2. The probability of having a decoder error is negligible with n — >- 00 if 

R 2 <C p , D (W), (22) 

wfrm? C p * D (W) = H(W * p * D) - H(p * D) = H(X) - H(p * D) z's f/ze cflpdfy o/f/ze BSC 
between Y and U, subject to a Hamming-cost constraint W such that Y + Z + Q distributes as X 
(i.e. the balls related to codewords ofC 2 are packed only over the space in which x n lies). 

The closure of the rate region where there are no errors is shown in Fig. 4; it is not empty for 
any q such that H(q * D) > H(p *D), and the minimum of Rq approaches the rate-distortion 
function R*(D) = H(p * D) - H(D) for all distortions < D < d c < p (Wyner & Ziv, 1976). 
In the Gaussian case (Z <G J\f(0, &%)), ^0 is generated from the distribution J\f(0, 0%) and again 

the scale factor oc = y 1 — D /erf is used. If R2 and Kq + ^2 are high enough, dithered quan- 
tization of ccX over the superposition of codes C 2 and Cq leads asymptotically to the rela- 
tion ocX = Cq -\- C 2 — Q, with Q ~ A/"(0,D) independent from Cq and C2. At the decoder, 
which knows eg, the channel between ocY and C2 satisfies C2 = ocY -\- ocZ -\- Q — Cq, i.e. it 
is Gaussian, with capacity C(W) = (1/2) log(l + W/(<x 2 (T% + D)) = (1/2) log(l + W/crf) 
under the quadratic cost constraint W. There are no decoding errors if the code rate R 2 is 
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below this capacity, and, again, the best estimate of the source at the decoder can be found 
by taking oc(cq + C2) + (1 — o?)y. The minimum achievable rate can be in this case com- 
puted with a geometric argument: the goal is to cover the n-dimensional balls related to 
codewords of C2 with as least as possible balls of average quadratic-distortion D. But the 
quadratic-distortion of the formers cannot be smaller than 0% so that the minimum code rate 
of code Co, provided that 0% + D > cr\, equals the rate-distortion function of the SCSI problem 
R*(D) = (l/2)log(Vf/D) (Wyner & Ziv, 1976). 

Despite in principle the decoder /encoder for a CCSI problem can be used as the en- 
coder/decoder of a dual SCSI problem (Pradhan et al., 2003), from a practical point of view 
CCSI and SCSI are very different. In CCSI source coding is done at the encoder; the code C2 
must be structured such that exhaustive search can be employed in order to perform a closest 
neighbor search (e.g. be a trellis code). Channel decoding is done at the decoder, and both 
Co and C2 should be performing channel codes for which a good algorithm approximating 
maximum likelihood search exist (e.g. turbo codes). In practice, no penalty is introduced in 
having C2 not very good from a channel coding perspective (Bennatan et al., 2006). 
In SCSI channel decoding is done at the decoder; C2 should be a performing channel code for 
which a good algorithm approximating maximum likelihood search exist (e.g. turbo codes). 
Source coding is done at the encoder, and the code Co + C2 must be structured such that ex- 
haustive search can be employed in order to perform a closest neighbor search (e.g. be a trellis 
code). Unfortunately, there are currently no algorithms that perform similarly to a closest 
neighbor search over good codes for channel coding. Hence, in Section 5 we will actually rely 
on convolutional codes Ci, paying in this case a penalty with respect to the theoretical bounds. 

4. Practical Iterative Algorithms for Lossless Coding 

Turbo codes are good practical channel codes, and hence are good tools for lossless (binary) 
SCSI. In the following we discuss the utilization of standard and ready-available turbo encod- 
ing and decoding algorithms for this problem. Differently from other contributions on this 
subject, we use the factor graph-based approach commonly taken in the LDPC-codes-related 
literature. For a useful tutorial article on factor graphs and message-passing algorithms, the 
reader is referred to (Kschischang et al., 2001). Under a unified formulation, we describe in 
principle the cases in which syndromes or parity bits are sent to the decoder, over binary or 
non-binary, lossless or lossy transmission channels, with binary or non-binary side informa- 
tion; more details can be found in (Cappellari & De Giusti, 2008). 

4.1 Turbo Codes Review and the Parity-Based Approach 

Turbo codes actually include different kinds of codes. In the most common case (parallel con- 
catenated convolutional codes) they are systematic codes: in correspondence of a sequence of Nk 
outcomes from X (x) the turbo encoder uses two systematic (n,k) convolutional codes to form 
two sequences of parity bits of N (n — k) + Zt bits each 4 (po and pi), according to the following 
algorithm. 



function TrbEnc(x) 

po <- GetParity (x) 

x' <- iNTERLEAVE(x) 



4 The additional z* <C N(n — k) bits are emitted while terminating the encoding into the zero state (zero- 
tailing). 
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P! <- GetParity 1 (x / ) 
return [x,p ,pi] 
end function 



Once x and p z are sent over a channel, they are received as r and r z - respectively, and the turbo 
decoder aims to maximize p rrori (x) ex / rrori (x)p(x). 5 The maximum likelihood (ML) decoding 
procedure is approximated by a message-passing algorithm on the factor graph of / rrori (x). In 
particular, Z r r ri(x) factorizes into lr(x)lr {x)l ri (x), and /r r (x) = Ep ; /Cz'(pz'l x )^(pz)/ where p z 
are the true parity sequences and ^ z (p z |x) are the indicator functions that are unitary if and 
only if pi is the parity of x (according to the z-th convolutional code, comprehensive of the 
interleaver for i = 1). The traditional decoding algorithm operates (on the factor graph of 
Fig. 5(a)) as follows, where the function FBA Z (-) computes the APP function relative to the 
z-th convolutional code, assuming q(x) as the prior probability and using the forward-backward 
algorithm (Bahl et al., 1974). If the prior probability p(x) is known, it can be absorbed into / r (x) 
in order to implement maximum a posteriori probability (MAP) decoding. 

1: function TRBDEC(/ r (x),/ ro (p ), Z ri (pi),M) 

2: m <— 0, i <— 1, initialize ^(x) > Initialization 

3: repeat > Turbo loop 

4: m<— m + 1, i<— 1 — i 

5: apM(x) ^FBA f <i r (x),i r< (p f -)/?(x)) 

6: q(x)^apM(x)/[l t (x)q(x)] 

until m > M 

return ap( m \x) 
end function 

The application of turbo codes to the SCSI problem X = Y + Z, with Z ~ B(p) and Y ~ 
8(1/2), is very straightforward. In particular, the parities are sent, and decoding is done 
by simply invoking TRBDEC(/ y (x), Z ro (po)/ Mpi), M ) (Aaron & Girod, 2002; Garcia-Frias & 
Zhao, 2001), where / y (x) = p zN k(x — y) takes into account for the virtual channel statistics 
(see Fig. 5(a)). In addition, it is possible to jointly decode and estimate p with no performance 
loss (Garcia-Frias & Zhao, 2001). Since puncturing (i.e. bit removal) can be employed at the 
encoder before transmission (the decoder can take into account this fact by assuming uniform 

2(n— k) 

likelihoods in correspondence of the punctured bits), any rate < R < -^ — '- is achievable. 

4.2 Syndrome-Based Approach 

A turbo code can be seen as a (N(2n — k) + 2zt,Nk) systematic block code whose generator 
matrix is G = [In* | Pol Pi]/ where P z - is the Nk x [N(n — k) + zt] parity formation matrix corre- 
sponding to the z-th convolutional code (comprehensive of the possible interleaver). If punc- 
turing is employed (exclusively on the parity bits), then the equivalent generator matrix is 
G ; = [I]^|Pq|P^], where Pj is the Nk x s z - matrix obtained removing from P z - the columns 
corresponding to the punctured parity bits. Consequently, 

H , = r gg I igg 1 0s x Sl l (23) 



5 Given two r.v. A and B, the likelihood and a posteriori probability (APP) functions will hereafter be denoted 
by l a {b) = p(a\b) and p a (b) = p(b\a) respectively. 
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pOiIpi) 



(a) parity-based approach (with p(r\x) — p(y\x)) 



p(xi|yi) 




(b) syndrome-based approach 
Fig. 5. Factor graphs representing the APP functions in the SCSI problem using turbo codes. 



is a parity-check matrix of the turbo code. In correspondence of a sequence of Nk + sq + si 
outcomes from X (partitioned into the three sub-sequences x, xq, and xi of length Nk, Sq, and 
Si respectively), the syndrome is obtained according to the following algorithm. Eventually, 

any rate < R < ^-fc < ^ bit/sample is achievable; for example, if (2, 1) constituent codes 
are employed, < R < 2/3 bit /sample. 



function SynEnc(x, xq, xi) 
[x,p ,pi] <- TrbEnc(x) 
for i <— 0,1 do 

p ^- PUNCTURE z (p z ) 
Sj <~ P + x z - 
end for 
return [sq,Si] 
end function 



We now directly tackle the problem of optimal MAP syndrome decoding, assuming that s z - 
are sent over a general channel and are received as r z . In particular, the decoding algo- 
rithm is derived from examining the factor graph of the APP function pyy y irori (xxoXi) ex 
Pyy oyi (xxoXi)Z rori (xxoXi). But pyy oyi (xxoxi) factorizes into p y (x)p yo (x )p yi (x 1 ); similarly 
/ rori (xxoXi) = / ro (xx )/ ri (xxi) / and/ r .(xx f ) = E Sj ^( s /) L Pi Xi(Pi\ x )X{p l+Xl = Sl }' where s ( and 
pi are the true syndrome sequences and the parity sequences from which they are computed 
respectively, and X{p i +x i =8 i } ^ s tne indicator function of the condition in brackets. The corre- 
sponding factor graph of the APP function is shown in Fig. 5(b). 

This factor graph is an extension without additional cycles of the factor graph relative to the 
parity-based approach. MAP decoding can be achieved by reusing the turbo decoding algo- 
rithm presented above; in particular, it is only necessary to form the correct input likelihoods 
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to the function TrbDec(-) and post-process the output APP function ap(x), for example using 
the following algorithm (hard syndrome decoding). 6 



function HSYNDEC(p y (x), p yo (x ), p yi (xi), l t0 (s ), Z ri (si), M) 
for i <*— 0, 1 do 

f;(p;) <- E S; ti(si) E X; ^( x i)^{ P; +x ; =s r } 

end for 

flp(x) 4r- TRBDEC(p y (x), / (po), *i(Pi)/ M) 
x <— argmax x ap(x) 
[x,p ,pi] <- TrbEnc(x) 
for i <— 0,1 do 

x 7 - <- argmax Xl . p yi (x z -)/ ri (p; + x,-) 
end for 

return [x, Xq,Xi] 
end function 



> Pre-processing 



> Post-processing 



Using the log-likelihood ratio and the log-APP ratio defined as 



h = log 



UP) 



A i P, 



'«(!) 



P«(0) ' 



(24) 



the y-th factor computed by the pre-processing operation (line 3) is lu = l t . . * p y . ., where the 
log-likelihood ratio convolution operator is defined as l\ * li = log e * "^ +^ 2 • Post-processing (line 
9) yields x 7 y = r z - v + p t ■ ,■; maximization translates into thresholding. 

The thresholding and parity formation operations permit the reutilization of the traditional 
turbo algorithms. Nevertheless, they prevent the computation of the correct messages across 
the nodes p z , so the hard syndrome decoding algorithm is not optimal. But if a full turbo 
decoding function FTrbDec(-) is used that outputs the APP functions ap(pj) for the parity 
bits as well, the post-processing can be improved as follows (soft syndrome decoding). 



function SSYNDEC(p y (x), p yo (x ), p yi (xi), / ro (s ), Zn(si), M) 
for i <*— 0, 1 do 

kiVi) <~ E S/ k( s i) E X/ Py l ( x i)X{ Pl+Xl =s l } 
end for 

[ap(x) / ap(p ) / ap(p 1 )\ <- FTRBDEC(p y (x), Z (po), *i(Pi)/ M) 
for i <*— 0, 1 do 

«p'(x,-) <- E S/ ^(Si) E p/ 7^f^{p,+x/=s ; } 

ap(x,-) i-py.ixfiap'fa) 
end for 

return [flp(x),flp(xo),flp(xi)] 
end function 



> Pre-processing 



> Post-processing 



Now, in order to estimate x z - ,• thresholding will be applied to ap Xij = p y . . + Z r . . * (fl/?p f ■ — // v), 
which is certainly more accurate than before. If the transmission channel is error-free 



' The operations in the pre- and post-processing (that do not involve the punctured parity bits) are writ- 
ten in the most general fashion, but in practice they are symbol-wise operations between marginal func- 
tions. 
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(i.e. |/ r .. | = oo ) it is easy to show that ap Xij equals ap Vij or —ap Vij if r z -y is or 1 respectively, 
because in either case the contributions of p y . . and In cancel out; then, x z y can be again esti- 
mated by Xj f j = ti f j + pi r j, where p 7/ y is the estimate of the corresponding parity bit obtained 
thresholding app.. rather than the one obtained invoking the function TrbEnc(x). On the 
contrary, if the TC is not error-free |/ r ..| < oo, and hence ap Xij must be actually computed in 
order to correctly estimate x z y. 

4.3 Experimental Results and Comparisons 

A virtual channel has been simulated with Y ~ 8(1/ 2) and Z ~ B(p), for different values of p; 
error-free transmission has been considered for comparison purposes with previous literature 
on this subject. Both parity- and syndrome-based approaches have been simulated. The turbo 
code uses two identical (n,k) = (2,1), 16-state, systematic constituent codes with generator 

matrix G(D) = 1 ^[XplxpP / L^J parity /syndrome bits are sent in correspondence to 

each data frame of L = 2 16 = 65536 samples. Puncturing is performed such that rates R of 2/3 
or 1/2 bit/samples are achieved. 2 13 = 8192 frames have been generated for each p, such that 
the average bit error ratio (BER) is eventually estimated over 2 29 ~ 5 • 10 8 bits. The decoding 
routines TrbDec and SSynDec are set for a number of runs of the FBA algorithm M equal to 
40. 

Comparisons are given in Fig. 6, in which the BER is shown as a function of H(p). When 
R = 2/3, the proposed method 65536-SSynDec outperforms the coding performance of the 
"SF+ISF" method given in (Tu et al., 2005). Despite the different syndrome formation proce- 
dure used in the latter (which does not rely on a standard turbo encoding engine), these two 
methods are very similar in the way they work. Hence, it is reasonable to think that the dif- 
ferent performance is the result of better coding parameters (i.e. frame length, convolutional 
code, interleaver and puncturer). Despite the very large interleaver length, the "Syn. trellis" 
method proposed in (Roumy et al., 2007) has very poor performance, which is even worse 
than the performance of the parity-based method 65536-TrbDec. When R = 1/2, the pro- 
posed method 65536-SSynDec has again a good performance, which are surpassed only by 
the LDPC-based systems reported in (Liveris et al., 2002) (which employ a longer frame size) 
and by the "P&C trellis" method proposed in (Liveris et al., 2003), which makes use of longer 
frames and of different 16-state constituent codes (specifically tailored for heavy data punctur- 
ing). Again, despite its smart formulation and very long frame size, the "Syn. trellis" method 
(Roumy et al., 2007) has very poor performance. 

5. Continuous-Valued Syndromes for Lossy Coding 

Traditional (discrete) syndromes of a linear code are good for lossless SCSI. In this section 
we discuss a coding method based on continuous-valued syndromes of a lattice for lossy SCSI 
of continuous sources with quadratic distortion (Cappellari & Mian, 2006b). This method 
embodies the superposition coding approach described in Section 3.2.1. 

Consider a lattice A C IR n . Being a subgroup of IR n , it induces the partition of IR n into the 
cosets of IR n /A. Each coset is uniquely identified by one of its elements; in particular we 
assume as coset leader of L e K n /A the element l(L) = argmin AeL d(\,0). We call continuous- 
valued syndrome (CVS) of x n e IR H , relative to A, the element s^(x n ) = l(L) such that x n e 
L. If we define the quantizer Qa( xH ) — argmin AeA d(x n , A) (with the further condition A + 
s a(*") — x n i n case °f ambiguity), the CVS satisfies s^(x n ) = x n — Qa(* h )- 
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(b) R = l/2 



Fig. 6. Comparison between different SCSI methods. The label "SF+ISF" refers to the 
syndrome-based method in (Tu et al., 2005) (results for two different convolutional codes are 
shown); the label "Syn. trellis" refers to the syndrome-based method in (Roumy et al., 2007), 
where 16-state constituent codes are employed. The label "Turbo parity" refers to the parity- 
based method in (Aaron & Girod, 2002), that uses two (5, 4) 16-state constituent codes. The 
label "LDPC" refers to the syndrome-based method in (Liveris et al., 2002) (results relative to 
two irregular LDPC codes are shown); the label "P&C trellis" refers to the syndrome-based 
method in (Liveris et al., 2003) that uses 16-state constituent codes. The frame length is re- 
ported too. 



We consider the additive virtual channel X = Y + Z. If P[Q A (Z n ) ^ 0] is asymptotically 
negligible (i.e. the realizations of Z n lie in the fundamental Voronoi region of A), then s^(x n ) 
permits near lossless reconstruction at the decoder; in fact 



x n =y n +z n (!) y » +Sa ( 2 «) ® f + Sa ( Sa(x ») + s A {-f)) 



(25) 



where ( x ) holds with high probability and ( 2 ) follows from linear properties of the CVS. In 
practice, the reconstruction from s^(x n ) and y n is obtained by a single quantization as 



x" = s A (x n )-Q A (s A (x")-y") . 



(26) 



Of course, we would need a channel with infinite capacity in order to transmit the CVS to 
the decoder. Hence, we assume that a quantized version Sa^(x n ) of the syndrome sa A (x n ), 
such that S«a — Sra ~ Q with Q independent from Sa A , is actually transmitted by the en- 
coder. 7 For j6 > 1, the realizations of Z n and, reasonably, the ones of the error Q n , lie in the 
fundamental Voronoi region of jSA. The reconstruction, according to (25), satisfies 

*" = y n +s^{spA{x n )+s^ A (q n )+s^ A (-y n ))=y n +s^ A {z n +q n ) (27) 

= *»+(4»-Q,$ A (2» + 0)=*" + (l n + fil)> (28) 



7 If the Voronoi regions of /3 A are asymptotically spherical, Sp A and Q exist that are approximately Gaus- 
sian. 
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Fig. 7. Wyner-Ziv coding using continuous-valued syndromes. 



where Q j is the overload error. If Qt = Q + Q \ is independent from Y and Z (and the mean 
of all random variables is zero), Y — » X — >- X forms a Markov chain, such that the minimum 
mean square error linear estimate for X computed at the decoder is eventually given by 



X 



-X + 



<r£ + o~i of + <ri 



-Y , 



(29) 



1/cr 1 + 1/(7?, where a? 



E[Z 2 } and ^ 



and the achieved distortion D satisfies 1/D = 

E[Q 2 }. This coding scheme is shown in Fig. 7. 

For a fixed transmission rate R the best possible syndrome is sent, and the parameter ft is 

experimentally tuned in order to minimize the variance of D. In theory, in the Gaussian case 

(Z ~ Af(Q,o~ 2 )) with A being a good channel code for the virtual AWGN channel between 

Y and X, ft = 1/a = 1/yl- D*(jR)/(7f guarantees that the rate-distortion function R*(D) 

is achieved (D*(jR) = o~ 2 2~ 2R is its inverse). In fact, in this case, (i) since the syndrome dis- 
tributes uniformly over the fundamental Voronoi region of ft A and A is a good channel code 
for the noise Z, S«a ~ A/"(0, ft 2 o~ 2 ), (ii) the minimum distortion in sending the syndrome at 
rate R is such that E[Q 2 ] = ft 2 a 2 2~ 2R = ft 2 D*(R), (iii) since a 2 + E[Q 2 ] = ft 2 a 2 and ^A is a 
good channel code for an AWGN of power ft 2 o~ 2 , there is no overload error, i.e. a 2 = E[Q 2 }, 
(iv) D = a 2 z E[Q 2 ]/(al + E[Q 2 ]) = £[Q 2 ]//5 2 = D*(R). 

A superposition of two codes is indeed used where the coarse code is the lattice code ft A and 
the additive code is the source code used for quantization of Sa^. The encoder is dual to 
the decoder of a multiple access channel problem that operates by interference cancellation: first, 
the codeword of the code with the lower code rate (the coarse one) is computed; then, this 
is subtracted from the source and the codeword to be transmitted to the decoder is formed 
according to the additive code. In practice, since we must be able to conduct an exact closest 
neighbor search on the coarse code, it cannot be a very performing channel code, i.e. we must 
cope in general with an overload error Q / (which corresponds to having a decoder error, as 
defined in Section 3.2.1). The parameter ft permits to balance the contributions of the two 
errors. The higher is ft, the lower is the overload error; but, for fixed transmission rate R, the 
higher is ft, the higher is the variance of the granular error Q. 



5.1 Experimental Results and Comparisons 

We simulated the AWGN channel X = Y + Z with a Gaussian input Y ~ Af(0, oi = 1) and 
a noise Z with various variances a 2 . We employ trellis-coded quantization (TCQ) based on the 
partition aZ/4aZ for syndrome formation (Marcellin & Fisher, 1990), which defines the lattice 
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(a) effect on the errors 
Fig. 8. Optimization of the volumetric parameter /3 2 . 




(b) optimum value 



A; the length of each input data frame is n = 1000; a is tuned such that the experimental 
second moment per dimension of the Voronoi regions of A equals a 2 . Any suitable source 
coding algorithm can be used to code the n-dimensional CVS; again, we employ TCQ, based 
on the partition frZ/4frZ and on 8-state trellises. For each rate R, b is tuned such that the 
normalized volume of the Voronoi regions induced by this quantizer is 2 2R times less than the 
corresponding parameter of /3A. Actually, in the following, R denotes the average entropy 
for describing the quantized syndrome, measured with the 2-supersets context-based method 
proposed in (Marcellin, 1994). 

The effect of ft 2 on the variances of the granular and of the overload error is shown in Fig. 8(a), 
for (Ty/cr 2 = 14.0 dB and at two different target rates. At low /3 2 , the total error (solid curves) 
is approximated by the overload error (dashed curves); at high /3 2 , the total error is approxi- 
mated by the granular error (dot-dashed curves). The circles indicate the optimum value of 
j6 2 and the corresponding distortion. The optimum j6 2 does not depend on the variance ratio 
cr 2 /cr 2 but only on the coding rate R of the system. In particular, this optimum is shown in 
Fig. 8(b) and it is decreasing with the rate. By increasing the number of states of the trellis, A 
gets closer to a good channel code and consequently ft 2 decreases; for comparison, the figure 
also shows 1/a: 2 , i.e. the lower bound for j6 2 . 

The experimental performance loss with respect to the distortion-rate function D* (R) is shown 
in Fig. 9(a). More precisely, the measurements are relative to a CVS-based system in which the 
TCQ used for syndrome quantization has been optimized with an algorithm adapted from 
(Chou et al., 1989). Similarly to the optimum value of /3, the performance loss does not de- 
pend on cr 2 /cr 2 ; for each rate, the value shown is the average over the various values of o~ 2 
(o~y/cr 2 is in the range 9 -f- 19 dB). The error bars show the average of the 95 % confidence 
intervals; for each value of a 2 the confidence interval is estimated over 5000 independent sim- 
ulations. The experiments show that the performance of the proposed system is within 3-1-4 
dB from the theoretical bound at rates between 0.5 and 3.0 bit/sample. A comparison is given 
in Fig. 9(b) against another practical SCSI system, namely the "DISCUS" system (Pradhan & 
Ramchandran, 2003). By simply choosing the right scaling factor a, tuned with respect to a 2 , 
the proposed system adapts to any correlation and gives the same performance loss, while 
the "DISCUS" system should be optimized for different correlations. This would not be an 
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(a) performance loss 
Fig. 9. Experimental performance of CVS-based SCSI. 
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easy task since it would involve the redesign of a source and of a channel code. Moreover, 
while only integer rates can be achieved by the "DISCUS" system, any rate can be achieved 
by SCSI based on CVS. This can be simply obtained by choosing the right value of b in case of 
syndrome coding with TCQ or using another ad-hoc source coding method for transmitting 

The proposed coding method turns out to be competitive with the SCSI methods in which the 
channel code component is a convolutional code; in addition, it allows for easy adaptation to 
the virtual channel statistics and to the desired transmission rate. More involved decoding 
algorithms for the actual case in which the virtual channel is not exactly known, and appli- 
cations of this coding scheme in the video coding scenario have been proposed too and are 
briefly discussed in the following. 

5.2 Iterative Algorithms for Unknown Virtual Channels 

The factor graph approach used in Section 4 for lossless SCSI turns out to be useful for op- 
timized CVS decoding too, in place of the simple operation given in (26). In particular, in 
(Cappellari, 2008) a factor graph-based decoding method is discussed for the case where the 
virtual channel statistics is time-varying and not exactly known at the encoder. More precisely, 
with the hypothesis of a negligible overload error such that X « Y + Z, where Z = Z + Q 
is independent from Y , a doubly stochastic hidden Markov model (HMM) (Rabiner, 1989) is as- 
sumed for Z. The model has L states and the distribution corresponding to the ;-th state is the 
generalized Gaussian distribution (GGD) Goc(fi,0f). The simulated n-dimensional realizations 
of Z n are identified within another HMM with possibly different number of states and state 
variances in order to simulate a partial knowledge regarding the virtual channel; this infor- 
mation is used by the encoder and transmitted to the decoder (the required bit-rate is taken 
into account) for syndrome formation and decoding, respectively. 

Since S = S + Q (we will omit subscripts for clarity) is with good approximation the syndrome 
corresponding to X = X + Q, Y ^ X ^ S forms a Markov chain, and optimal MAP decoding 
amounts to maximizing 

f(x n \y",s")c<f(x n \y n )f(s n \x n ), (30) 
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f(x n \y n ) 




Fig. 10. Factor graph-based CVS decoding (n = 4). 



f(s n \x n ) 



where the terms f(x n \y n ) = fj(x n — y n ) and f(s n \x n ) take account of the virtual channel 
structure and of the syndrome formation algorithm, respectively. Once we define the hidden 

state variables of , the state transition probabilities p (of \of_^ ), and the state probability densities 



cc/2 



fz( a \J) = - 

y/2T(l/a) t? 
marginalizing 



exp 



\ a -Vj\ 



, with yj = ofT (l/a)/2T (3/a), f% is found by 



fr (z»;^) = p (o£)/ 2 (z |^) n p (erf \of_ x )f t [z x \af) 



(31) 



The (TCQ-based) syndrome formation is instead a deterministic transformation, i.e. f(s n \x n ) 
is a Dime's delta function that, given x n , reveals its syndrome. Equivalently, it is a delta 
function that reveals the event {x n — s n G jSA}; by introducing the trellis state variables of, 
f(s n \x n ) is found marginalizing 



m—\ 



f{$ n ;o- c \xA = i\ x y E s(x k -s k -b) / 

k=0 k ~ l af 



(32) 



beB ' 



where x) an ^ &\ are the indicator function and the set of reconstruction values relative to the 
transition from state j to /, respectively. 

In order to increase the independence between the information about X brought by Y and by 
S, the syndrome formation algorithm is operated on a randomly scrambled version of x n , such 
that the actual factor graph used in CVS decoding is similar to the one sketched in Fig. 10. At 

the m-th iteration, model messages ]i^ (xi) and code messages ]Iq (x-^) are exchanged between 
the upper and the lower part of the factor-graph and viceversa, until they converge to a fixed 
value. The final estimate x n for x n is given by 



x { = argmaxf{xi\y n )f(s n \xi) . 



(33) 



We report some experimental results for the case where Y ~ A/"(0,c^ = 1) and Z is a 3-state 
Gaussian Markov process {oc = 2). The distortion-rate function obtained with traditional ML 
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Fig. 11. Experimental results for factor graph-based CVS decoding; Di(R) is an estimate of 
the theoretical distortion-rate function of the considered SCSI problem. 



decoding, with MAP decoding (i.e. considering only one iteration in the decoding process), 
and with TURBO decoding (i.e. at convergence) are measured. The syndrome is uniformly 
quantized and entropy coded for transmission at the decoder; 1000 sequences of n = 1000 
samples each are generated. If the encoder has knowledge about the virtual channel statistics 
(i.e. the same model is artificially used for hidden state identification 8 ), the TURBO decoding 
algorithm does not perform any better than the MAP decoding algorithm, that in turn was 
shown to perform like the ML decoding (at least in the Gaussian case). If a single-state HMM 
is used for hidden state identification (i.e. the encoder only knows the average variance of Z), 
the MAP decoding performs again like the ML decoding, but TURBO decoding leads to about 
one order of magnitude less reconstruction errors, which translate into an up to 3 dB decrease 
of the mean error variance (see Fig. 11) at bit-rates R = 3 -=- 4 bit/sample. 

5.3 Video Coding Applications 

Traditional video coding standards, e.g. H.264/AVC (ITU-T & ISO/IEC: JTC1/SC29/WG11, 
2007), are based on predictive coding for exploiting the high temporal correlation between ad- 
jacent frames. This implies that (i) the algorithms used during encoding are computationally 
heavy with respect to the ones used at the decoder, and (ii) the coded representation is very 
sensible to possible packet losses on the transmission channel. To alleviate these problems, in 
order to permit effective video encoding and transmission on wireless and battery-operated 
devices, several research groups have recently explored SCSI-based methods for video com- 
pression. A review on these methods can be found in (Girod et al., 2005). 
In practice, a frame is encoded assuming that some adjacent frames are already available as 
SI at the decoder. Hence, (i) the encoding algorithm is more light since it does not exploit the 
inter-frame correlation, and (ii) packet losses are not so bad as long as several adjacent frames 
are stored at the decoder that can be used as SI. 

The CVS method has been applied to video coding in (Cappellari, 2007; Cappellari & Mian, 
2006a). In particular, it has been tested in both the discrete cosine transform (DCT) and the dis- 
crete wavelet transform (DWT) domain. In both cases, every other frame is sent as an intra-frame 
(without referencing any adjacent frame); the remaining ones are sent as inter-frames. The 
decoder performs motion compensated interpolation for each couple of consecutive intra-frames 



! There may be still uncertainties about the actual hidden states. 
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Fig. 12. Scheme of the proposed DWT-domain CVS-based video coding method and perfor- 
mance of different video codecs, averaged on the first 100 frames of the sequence foreman 
(QCIF resolution) at 25 frames per second (data relative to intra frames are not taken into 
account). 



in order to construct the SI for decoding the inter-frame between them. In the DCT case, inter- 
frames are partitioned into 8x8 blocks of pixels and the DCT coefficients corresponding to the 
lower spatial frequencies of each block form the signal to be coded relying on co-positioned 
coefficients of the SI; the remaining coefficients are sent in intra mode. In the DWT case (see 
Fig. 12(a)), the DWT coefficients are classified into the classes intra, inter, and skip, for coef- 
ficients that are expected to have little, medium and high correlation with the co-positioned 
coefficients of the SI, respectively. Intra coefficients are sent in intra mode, no information is 
sent for the skip coefficients (the corresponding SI is taken directly as reconstruction), and the 
remaining ones form the signal to be coded using SI. 

Classification and correlation estimation are in both cases performed using the block-based 
MSE between the inter-frame to be coded and the previous one. These estimates are used 
for proper TCQ-based syndrome formation and decoding. Syndromes are quantized with an 
embedded uniform quantizer, such that quality scalable reconstruction is achieved at the decoder. 
The typical performance of the proposed coders is shown in Fig. 12(b) and compared with 
results from (Aaron et al., 2004); the average peak signal-to-noise ratio (PSNR) is reported as a 
function of the average transmission rate. The DWT-CVS coder outperforms both the DCT- 
CVS coder and one of the coders (Ave-I) from (Aaron et al., 2004), while being very close to the 
other solution (MC-I). The superiority of this (turbo code-based) method is probably due to 
the utilization of a feedback channel between decoder and encoder for estimation of the actual 
Si-source correlation. This solution is very good for this purpose, but also highly unpractical 
with respect to correlation estimation at the encoder only. 

6. Conclusion and Future Research 

In this chapter, we presented the SCSI problem and discussed several practical solutions. Our 
main contribution is the coding method based on continuous-valued syndromes, which is an 
embodiment of the theoretically optimal superposition coding approach. We showed that this 
coding method is very practical due to the separation between channel and source coding. 
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In particular, we showed that it can be quite performing in the actual scenarios, where for 
example the actual source-SI correlation is not exactly known and /or very complex, as in the 
case of video coding. 

In practice, despite its optimality, the superposition coding approach for SCSI suffers some 
performance loss because there are currently no efficient algorithms for source coding over 
codes with a " random structure " . Indeed, a good channel code should be used as coarse code, 
but all good channel codes have this "random structure" (e.g. turbo codes rely on random 
interleaving). Convolutional codes are less performing but are still the best ones for which a 
good closest neighbor search algorithm exists. In the future, it may be possible that message- 
passing algorithms will be developed that permit quantization over sparse codes that are good 
for channel coding; some effort in this direction is discussed in (Ciliberti et al., 2005; Martinian 
& Yedidia, 2003). 

In the near future, we plan to be more concerned with the problem of correlation estimation 
in the actual SCSI scenarios. In fact, all SCSI schemes proposed in literature are usually in- 
vestigated under the hypothesis of a toy virtual channel Y = X + Z in which the statistics of 
Z is known at the encoder and at the decoder. But in practice the source-SI correlation is not 
known and is more complex, so that by using these simple assumptions we incur into some 
performance degradation. For example, the performance of the SCSI-based video coders is 
still under the one of the traditional coders, at least in the scenarios with no losses on the 
transmission channel. We hope that our investigative efforts into statistical model aided de- 
coding could be eventually used towards improving the efficacy of SCSI-based coding not 
only in video coding applications, but also in several other practical cases. 
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1. Introduction 

Sensor array technique has been widely used for measuring various types of wavefields such 
as acoustic waves, mechanical vibrations, and electromagnetic waves (1). A common goal of 
array signal processing is estimating locations of sources or separating source signals based 
on multiple observations. For obtaining efficient spatial information, the geometrical arrange- 
ment of sensors is one of the significant issues in this field. An uniform linear array is the most 
popular and fundamental one (2; 3), and suiting with purposes, various types of arrays have 
been considered such as circular, planar, cross-shaped, cylindrical, and spherical arrays. 
In this chapter, we discuss the sensor arrangements from a new viewpoint: correlation be- 
tween channels. Generally, multiply-observed signals have correlation each other, and it be- 
comes larger especially in a small-sized array. In the case, observed signals themselves are 
not efficient representation due to redundancy between channels. Although they are uncor- 
rected by appropriate basis transformation, which is corresponding to the diagonalization of 
the covariance matrix, it depends on the observed wavefield. 

However, in isotropic wavefield, there exist special geometrical sensor arrangements, and 
observed signals by them are commonly uncorrelated by a fixed basis transform. The signifi- 
cances of isotropic wavefield decorrelation are as follows. 

• If there is no a priori knowledge to wavefield, the isotropic assumption is simple and 
natural. It means spatial stationarity 

• It is well known that Fourier coefficients of a temporally stationary periodic signal are 
uncorrelated each other. The isotropic wavefield decorrelation can be considered as 
a spatial version of it and decorrelated components represent something like spatial 
spectra. 

• The decorrelated representation are also useful for encoding because redundancy be- 
tween channels is removed. 

• It can be applied for several kinds of estimation methods in isotropic noise field such as 
power spectrum estimation (4), noise reduction (5), and inverse filtering (6). 

• The isotropy assumption can be valid even if wavefield is disturbed by sensor array 
itself. Suppose that microphone array is mounted on a rigid sphere. Although the rigid 
sphere disturbs acoustic field, due to the symmetry of sphere, the isotropy is still hold. 
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Fig. 1. Square 



Although our main concern lies on microphone array, this technique can be applied for differ- 
ent kinds of wavefield sensing. In the following, we mathematically discuss possible sensor 
arrangements for blind decorrelation. 



2. Problem Formulation 

Let's consider isotropic wavefield is observed by M sensors. Let x m (t) be a signal observed 
by the rath sensor, X m (co) be its Fourier transform, and X(co) = (Xi(a?) X2(co) • • • Xm(cv)Y 
be the vector representation, respectively, where t denotes transpose operation. The isotropic 
assumption leads: 1) the power spectrum is the same on each sensor, and 2) the cross spectrum 
is determined by only a distance between sensors. Under them, by normalizing diagonal 
elements to unit, the covariance matrix V(co) of the observation vector X(cv) is represented as 



/ 



V(a>) = E[X(cv)X(cv) h ] 



T(r 21 ,cv) 



T(r 12 ,cv) 
1 



V T(r nl ,cv) T(r n2 ,cv) 



T(r ln ,<v) \ 

T(r 2n ,cv) 



J 



(1) 



where £[•] denotes expectation operation, h denotes Hermite transpose, r« is the distance be- 
tween sensor i and j, and T(r,co) represents the spatial coherence function of the wavefield 
(3). Under the isotropic assumption, V(co) is a symmetry matrix since r z y = r^. Then, there 
exist an orthogonal matrix U for diagonalizing V(cv). Our goal here is to find special sensor 
arrangements and corresponding unitary matrices U such that U f V(a;)lI is constantly diag- 
onal for any coherence function T(r,co). We call this kind of decorrelation blind decorrelation 
because we don't have to know each element of V(co) and the diagonalization matrix U is 
determined by only sensor arrangements. For simplicity, we hereafter omit cv and represents 
the covariance matrix of the observation vector by just V. 

Intuitively, it seems to be impossible since a diagonalization matrix U generally depends on 
the elements of V. But suppose that four sensors are arrayed at vertices of a square. There 
are only two distances among the vertices in a square: one is the length of a line L, another is 
the length of a diagonal y/lh. Then, numbering sensors circularly shown in Fig. 1 and letting 
a = T(L,co) and b = T(\/2L,cv), the covariance matrix is represented as the following form 



V ■ 



/ 1 a b a \ 

a 1 a b 

b a 1 a 

\ a b a 1 ) 



(2) 
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for any to and any coherence function T(r,cv). Since it is a circulant matrix, it is diagonalized 
by the fourth order DFT matrix Z4 or its real-valued version Z4 defined by 
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This diagonalization can be performed at any frequency u> because Z4 is independent of a and 
b. It means the following basis-transformed observations: 



yi(0 = *l(*) + *2(*) + *3(*) + *4(0 

1/2(0 = X!(0-X 3 (0 

1/3 (0 = X 2 (t)-X 4 (t) 

1/4 (0 = X!(0-X 2 (0+^3 (0-^4(0 



(6) 

(7) 
(8) 
(9) 



are uncorrelated each other in any isotropic field. The problem we concern here is a general- 
ization of it. 
If IJtVU is diagonalized as 



utyu 



V is represented as 



V= U 
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Then, for blind decorrelation, one of the necessary conditions is that V is represented by only 
M parameters (71 • • • Jm) a * most. It means there should exist at most M kinds of distances be- 
tween sensors. Generally, when sensor arrangement has some symmetry, the number of kinds 
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Fig. 2. Argyle 



of distances between sensors is smaller. But what kind of symmetry the sensor arrangement 
should have for blind decorrelation is not trivial. For instance, suppose an argyle arrangement 
shown in Fig. 2. An argyle is one of symmetrical shapes and there are three kinds of distances 
among sensors. In arranging sensors shown in Fig. 2, the covariance matrix has the following 
form: 
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Despite of the symmetry of argyle, there are no matrices U for diagonalizing V in eq. (12) 
independent of a, b and c. It can be easily checked as the following (7). V in eq. (12) is 
decomposed as 

V = I + aP 1 + bP 2 + cP 3 (13) 

where I is an identity matrix and 
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For diagonalizing V by an unitary matrix U independently of a, b and c, it is necessary that 
Pi, P2 and P3 have to be jointly diagonalized, which is equivalent to the condition that P\, P 2 
and P3 are commutative each other. However, 



P1P2 ~ P2P1 



( 1 1 \ 

-10-10 

10 1 

\ -1 -1 / 



(17) 



which means P\ and P 2 are not commutative. Therefore, there are no matrices U to jointly 
diagonalize Pi and P 2 . More rigorous mathematical discussion is described in (7). 
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Note that the finding possible sensor arrangements for blind decorrelation includes two kinds 
of problems. One is what a matrix represented by several parameters is diagonalized inde- 
pendently of the values of the parameters, and the other is whether a corresponding sensor 
arrangement to the matrix exists or not. For example, 

/ 1 a a a a \ 

a 1 a a a 

V = a a 1 a a (18) 

a a a 1 a 

\ a a a a 1 / 

is diagonalized by the DFT matrix Z5 independently of a since V in eq. (18) is a kind of 
circulant matrix. However, eq. (18) means that each different pair of five sensors has the same 
distance, which cannot be realized in 3-D space. 



3. Crystal Arrays 

3.1 Necessary Condition 

First, we begin with the following lemma. 

Lemma 1. A necessary condition for V defined by eq. (1) to be diagonalized by an unitary matrix 
U for any function T(r,co), is that a set of distances from the sensor i to others: {r zl ,r z - 2 , • • • ,r/ n } is 
identical for any i. 

Proof: If V is diagonalized by an unitary matrix U without dependence on T(r, to), the matrix 
l n , of which all elements are identical to 1, is also diagonalized by U since I n is obtained by 
letting T(r,(v) = 1. Then, V and I n are commutative. From (i,j) element of VI n = I n V, we see 
that 

f^T{co,r ik )= £r(o;,r ;1 ) (19) 

fc=l fc=l 

has to be an identical equation of r z ys. It means that a distance set: {r z y | j = 1, 2, • • • n) must be 
identical for any i. m 

A square arrangement surely satisfies Lemma 1 since a set of distances from the sensor i to 
others is represented as {0, L, L, V^L}, which is identical to any i (i = 1,2,3,4). While, in an 
argyle arrangement, a set of distances is {0, L, L, D\\ from the sensor 1, and it is {0, L, L, D2} 
from the sensor 2. Thus, an argyle arrangement does r t satisfy Lemma 1. 

Lemma 1 directly gives a necessary condition of sensor arrangements for the blind decorre- 
lation, but it is not a sufficient condition. Actually, there exist arrangements which satisfies 
Lemma 1 but cannot be used for the blind decorrelation. An example is shown in Fig. 3. The 
shape is obtained by merging vertices of two triangles with the same center and a different 
angle in the same plane, denoted as a bi-triangle. 

In a bi-triangle arrangement, there are four kinds of distances between sensors: a short and a 
long line, and two kind of diagonals. The corresponding covariance matrix V is represented 
by 
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(20) 
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Fig. 3. Bi- triangle 




This arrangement obviously satisfies Lemma 1 since a set of distances from a sensor to others is 
identically represented as {0, L\, L 2 , L 2r D\, D 2/ D 2 }, but there is no matrices for diagonalizing 
U in eq. (20). 

Although it is not straightforward from lemma 1 to a specific sensor arrangement, we have 
found five classes of sensor arrangements for blind decorrelation up to now (4; 8). According 
to the geometrical resemblance with crystals, we call them crystal arrays. 

3.2 Five classes of crystal arrays 
1) Regular polygons 

Let circ denote a circulant matrix as 



circ{l,a,b) 




(21) 



In arraying sensors on vertices of a n-sided regular polygon, circularly numbering them as 
shown in Fig. 4 yields a circulant V = circ(l a\ a 2 • • • a 2 a{). As well known, it is diagonalized 
by n-th order DFT matrix Z n (9). Note that as a matrix to diagonalize V, we can choose a 
real-valued version of Z n as shown in eq. (4), instead of Z n itself, which leads simple basis 
transform in time domain discussed in section 2. 



© © 

Fig. 4. Regular polygons 




© © 

© © 



2) Rectangular 

The second class consists of only a rectangular. Under numbering sensors as shown in Fig. 5, 
V has a block-circulant structure as 



V 



h F 2 
F 2 F 1 



(22) 



where F\ and F 2 are 2x2 circulant matrices. It is diagonalized by U = Z 2 ® Z 2 - 
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Fig. 5. Rectangular 



3) Regular polygonal prisms 

The regular polygonal prism arrangement is given by merging vertices of two parallel n-sided 
polygons with the same center axis. As the rectangular case, V has a block-circulant structure 



V 



F 1 F 2 
F 2 F 1 

where F\ and F 2 are n x n circulant matrices. It is diagonalized by 



U = Z n <g> Z 2 



Zn 

Z n 



Zn 

~z n 



(23) 



(24) 



The two parallel n-sided polygon may have a certain different angle, which yields a twisted 
prism as shown in Fig. 6. In n = 2, any angles are allowable, which the matrix structure is 
invariant for. In n > 3, only the rotation with n/n is allowable, where V becomes simply 
circular by alternative numbering in the top and the bottom n-sided polygon as shown in Fig. 
6. 







Fig. 6. Regular polygonal prisms (upper) and their twisted versions (lower) 



4) Rectangular solid 

In related to a rectangular, a rectangular solid forms another class. By numbering sensors 
shown in Fig. 7, V has the following structure: 



V 



( h F 2 F 3 F 4 \ 

F 2 F\ F4 F3 

F 3 F 4 Fi F 2 

\ h F 3 F 2 F 1 ) 



(25) 
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where Fj (i = 1, 2, 3, 4) are 2x2 circulant matrices. V itself is not circulant but it has recursively 
circulant structure. Hence, it is diagonalized by U = Zi <g> Zi ® Zi. 




Fig. 7. A rectangular solid 



5) Regular polyhedrons 

As well known, there are only five polyhedrons in a 3D space: tetrahedron, octahedron, hex- 
ahedron, icosahedron, and dodecahedron, and they form the last class. From the viewpoint 
of the covariance matrix form, the tetrahedron is a special case of a twisted 2-sided polygo- 
nal prism, while the octahedron and the hexahedron are a special case of twisted 3-sided and 
4-sided polygonal prisms, respectively. The most difficult cases are given by the icosahedron 
and the dodecahedron arrangements. 






Fig. 8. Polyhedrons 



An icosahedron has twenty equilateral triangular faces. Let two opposed triangles be the 
top and the bottom faces. Then, all vertices lie in four parallel planes. Numbering vertices 
circularly in the top plane, and then, from the top to the bottom in order as shown in Fig. 8, 
we have 
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(26) 
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where 



F\ = circ(l a a), F 2 = circ(b a a), 
F3 = circ(a b b), F4 = circ(c b b), 
F5 = circ(l b b), F& = circ(c a a). 



(27) 
(28) 
(29) 



Unlike the other cases, V doesn't have the circulant structure. Taking into consideration that 
1) f< (1 < * < 6) is diagonalized by Z3 (the 3rd order DFT matrix) and 2) the block structure is 
different between the first, fourth columns and the second, third columns, we assume that U 
has the following form: 



U 



( 



\ 



Z 5 
Z3P3 
Z3P3 

z 3 



Z3 

Z3Q3 
Z3Q3 

Z3 



Z3 

-Z3R3 

Z3R3 

-z 3 



Z3 \ 


-Z3S3 


Z3S3 


-z 3 / 



where P3, Q3, K3, and S3 are diagonal matrices. Eq. (30) yields 

\ 



Z H VZ 



A 
O 
\ o 



A 
K 2 
O 
O 



O 
O 

K 3 
B 



O 
O 
B 
X 4 



where K,- (1 < i < 4) are diagonal matrices with the size of 3 x 3 and 

A = (G 1 + G2Q3 + G3Q3 + G 4 )+P 3 (G2 + G5Q 3 + G 6 Q3 + G 3 ) 
+P 3 (G 3 + G 6 Q 3 + G 5 Q 3 + G 2 ) + (G 4 + G3Q3 + G2Q3 + Gj) 

B = (Gi — G2S3 + G3S3 — G4) — -R3(G2 — G5S3 + G(,S3 — G3) 

+R 3 (G 3 - G 6 S 3 + G5S3 - G 2 ) - (G 4 - G3S3 + G2S3 - G x ), 

G\ = diag(l + la 1 — a I — a), 

G2 = diag(2fl + b b — a b — a), 

G3 = diag(fl + 2b a — b a — b), 

G 4 = diag(2fc + c c-b c-b), 

G 5 = diag(l + lb \-b 1 - V), 

G(, = diag(2a + c c — a c — a), 

where diag denote a diagonal matrix as 

diag(fl, b,c) = 
From A=0, we have 




2(1 + c)(l + vi<]i) + 2(« + b){2 + 3(pi + <7i) + 2pm) 

2(l + c-a-b)(l + p 2 q 2 ) 
2(l + c-a-b)(l + p 3 q 3 ) 



(30) 



(31) 



(32) 
(33) 

(34) 
(35) 
(36) 
(37) 
(38) 
(39) 



(40) 



(41) 
(42) 
(43) 



r 1 s 1 = -1, 


n + si = 4, 


risi = -1/ 


*"2 + $2 = 1, 


^3 = -1/ 


^3 + S3 = 1. 


"1=7+ + 7+/ 


si = 7- + 7 


^2 = *"3 = 7+/ 


S2 = s 3 = 7- 
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where p z - and qi (i = 1,2,3) are diagonal components of P3 and Q3, respectively. For satisfy- 
ing them for any ^ACfrACand c, there are ambiguities on determining pi, <\i, p 3/ q 3 since the 
conditions for them are only p^qi — P3^3 — 1/ Determining them the most simply, we choose 

Pi = P2 = P3 = h (44) 

m = qi = % = -1. (45) 

While, B = yields 

2(l-c)(l + risi)+2(a-fe)(2-(n+si)-2nsi) = 0, (46) 

2(l-c)(l + r 2 s 2 )-2(a-fc)(l-2(r 2 + s 2 )-r 2 s 2 ) = 0, (47) 

2(1 - c)(l + r 3 s 3 )-2(a-&)(l-2(r3 + s 3 ) - r 3 s 3 ) = 0, (48) 

where r z - and s z - (z = 1,2,3) are diagonal components of R 3 and S3, respectively In the same 
way as pi and qi, we have 

(49) 

(50) 

(51) 

Solving them, 

(52) 
(53) 

where r z - and s z - (i = 1,2,3) are diagonal components of R 3 and S3, respectively, and 

7+ = (1 + V5) /2, 7_ = (1 - \/5) /2. (54) 

Consequently, 

P 3 = diag(lll), (55) 

Q 3 = -diag(lll), (56) 

£3 = diag(7^_ + 7+ 7+ 7+), (57) 

S 3 = diag(7^+7_ 7_ 7_), (58) 

in eq. (30) gives us U to diagonalize eq. (26). 

By the similar numbering to the icosahedron shown in Fig. 8, V in the dodecahedron has the 

same block structure as eq. (26) where 

F\ = circ(l abba), F 2 = circ(a b c cb), (59) 

F 3 = circ(dcbbc), F4 = circ(e d c c d), (60) 

F5 = c\rc(lb ddb), F^ = circ(e c a a c) . (61) 

The form of U is also the same structure as eq. (30), just replacing the subscript 3 by 5, where 
P 5 = diag(l 7 2 _ <y 2 + <y 2 + 7 2 _), (62) 

Q 5 = -diag(l y 2 + 7 2 _ 7- 7 2 +), (63) 

R 5 = diag(7 2 f +7+ 7+ 7+ 7+ 7+), (64) 

S 5 = diag(7?.+7_ 7_ 7_ 7_ 7_). (65) 
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4. Conclusions 

In this paper, we discussed geometrical sensor arrangements for the blind decorrelation of 
isotropic wavefield. Based on a necessary condition, we showed specific five classes of sensor 
arrangements: 1) regular polygons, 2) rectangular, 3) regular polygonal prisms, 4) rectangular 
solid, and 5) polyhedrons, the first two of which have two dimensional, and other three have 
three dimensional geometries, respectively Specific orthogonal matrices corresponding to the 
sensor arrangements are also derived. 

Finding all possible sensor arrangements for blind decorrelation is still an open problem and 
we are investigating the relationship with the group theory in mathematics, especially, a point 
group (10). 
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1. Introduction 

In recent years, signal matching has been required in many fields. A number of matching 
methods have been developed, and an appropriate method should be selected for each ap- 
plication in order to obtain the desired performance (1)(2). Phase-only correlation (POC), 
phase correlation or PHAse Transform (PHAT) (3)-(17), which is referred to herein as POC, 
is a phase-based correlation that is used for various applications, such as delay estimation 
(3) (4), motion estimation (5), registration (6) (7), video detection (8) (9), and biometrics authen- 
tication (10) (11). Phase-only correlation with Fourier transform was developed as PHAT in 
sound /sonar processing literature (3), and POC with discrete Fourier transform was proposed 
by Kuglin and Hines (12). The concept of POC is based on the fact that the information related 
to the displacement of two signals resides in the phase of the cross spectrum. Combining POC 
with various techniques, such as interpolation and curve fitting, provides highly accurate es- 
timation (13)-(17). In special cases, the normalized cross spectrum corresponds to the product 
of the signs of discrete cosine transform (DCT) coefficients. Previously, we derived this rela- 
tionship mathematically and proposed DCT sign phase correlation (DCT-SPC) based on this 
relationship (18). DCT-SPC is a phase-based correlation and has properties that are similar to 
those of POC. 

Images, particularly in the fields of biometrics, medicine, and surveillance camera require 
extreme security in order to avoid the risk of identity theft and invasion of privacy (19). Gen- 
erally, encrypting and scrambling are used to protect information (20) (21). However, these 
protected images require decrypting or descrambling before image matching. In other words, 
neither POC nor DCT-SPC can be directly applied to conventional encrypted and scrambled 
images. Based on privacy concerns, secure multi-party techniques were applied to vision 
algorithms such as Blind Vision in (22). However, in (22), neither the registration nor the 
estimation of the geometric relationship between two images was discussed. 
In this chapter, for POC and DCT-SPC, we present phase-scrambled signals and a match- 
ing method that can be directly applied to phase-scrambled signals without descrambling. 
The presented methods are motivated by secure data management. The phase scrambling 
distorts only the phase information, which contains significant information of signals. Phase 
scrambling protects against the exposure of the information in the signal. Synchronized phase 
scrambling yields the relationship between non-scrambled signals. Therefore, POC and DCT- 
SPC can be directly applied to phase-scrambled signals. Moreover, the presented scrambling 
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(a) conventional templates (b) secure templates by phase scrambling. 

Fig. 1. Stored templates for phase-based correlation, (a) Conventional templates: If the tem- 
plates in a database were to be stolen, information about the original images would be vulner- 
able, (b) Secure templates: A template is stored in either a phase-scrambled coefficients form 
or a scrambled phase information form in order to guarantee secure data management. Phase 
scrambling prevents templates from putting at risk the information about the original images. 



has no effect on image matching. That is, the same accuracy is obtained from phase-scrambled 
signals without descrambling (23) (24). 

This chapter is organized as follows. In Section 2, we describe the motivations and impor- 
tant considerations of the present study. In Section 3, POC and DCT-SPC are explained. In 
Section 4, the phase-scrambled signals and image matching for POC are described. We ex- 
plain the reason why the POC between phase-scrambled signals has the same accuracy as 
that between the non-scrambled signals. In Section 5, the sign phase-scrambled signals and 
image matching for DCT-SPC are described. In Section 6, various simulations are presented 
for the purpose of confirming the effectiveness and appropriateness of the scrambled signals 
and image matching. Finally, Section 7 concludes this chapter. 



2. Image matching between visually protected images 

Image matching for authentication requires several templates that have been registered previ- 
ously. Generally, the management of these templates requires a great deal of labor. Counter- 
measures to prevent theft and refusal of cross-references 1 are required. Phase-based correla- 
tion uses the phase information of signals. Specifically, POC and DCT-SPC require the phase 



1 Templates registered in a particular system being diverted to another system without the permission of 
a registrant. 
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Fig. 2. Model of template-generation and matching algorithms for secure data management 



factors of DFT coefficients (DFT phase factors) and the signs of DCT coefficients (DCT signs) 
respectively, for translation estimation. In addition, both correlations require the magnitude 
of DFT coefficients for rotation and scaling estimation (6). When the effect of rotation and 
scaling are small and can be ignored, only phase information is used for matching. Therefore, 
the conventional template for phase-based correlation is stored in either the coefficients in the 
transformed domain or in a phase information form. If the template stored in the coefficients 
were to be stolen, the information in the original signal may be compromised. Even in the case 
of the templates stored in the phase information form, the information in the original signal 
may be exposed by the inverse transform of the phase information, as shown in Fig. 1 (a). In 
addition, neither the templates stored in the coefficients form nor the phase information form 
is considered for cross-referencing. Moreover, the templates stored in either the coefficients 
form or the phase information form may be modified or removed. Alternately, new templates 
may be introduced to the database. In order to address these problems, we focus on secure 
data management. 

In this chapter, we present phase-scrambled signals and a matching method for these sig- 
nals using POC and DCT-SPC. The presented method is motivated by the need to guarantee 
secure data management. The template is stored in either phase-scrambled coefficients or 
a scrambled phase information form, as shown in Fig. 1 (b), and the complete information 
about the original signal is protected by phase scrambling. In addition, the templates are 
used for image matching without descrambling. Synchronized scrambling by the same key 
allows estimation of the translated, rotated, and scaled values between an image and the tem- 
plate by phase-based correlation. Desynchronized scrambling using different keys prevents 
cross-referencing of templates, thereby guaranteeing secure data management. Note that, in 
the presented method, phase scrambling has no effect on matching using POC and DCT-SPC. 
That is, the estimation value between the phase-scrambled signals is obtained with the same 
accuracy as that between non-scrambled signals. 
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3. Phase-based correlation 

Two phase-based correlations, POC and DCT-SPC, are explained. Single-dimensional nota- 
tion is used for the sake of brevity. Let C, IR, and Z denote the sets of complex, real, and 
integer numbers, respectively. 

3.1 Phase-only correlation (POC) 

Let the N-point DFT of the N-point real signal #(n), (i = 1,2) (n = 0, 1, • • • N - 1) be G^k), 
(k = 0, 1, • • • , N — 1). Gf(k) is expressed in polar form as 

G,-(fc) = |G,-(fc)|e^ (1) 

= |G,-(*)|?G,(k) (2) 

where / = v 7 — T- The quantities | G z - (k) | and 0^ are the magnitude and phase, respectively. 
(pQ. (k) = ei° ik is referred to as the phase factor. 
The normalized cross spectrum is given as 

R tp (k)=4> Gl (k)-4>* G2 (k), (3) 

where <pQ (k) denotes the complex conjugate of (pc 2 (k). 

The POC is defined as the inverse DFT of Ify(fc) in (10)-(12), i.e., 



1 



N-l 



^(n) = -^R (p (k)W N nk / n = 0,l,...,N-l, (4) 



N 



k=o 



where W^ denotes e i 2n/N . The integer displacement value between signals is estimated 
using (4). 

3.2 DCT sign phase correlation (DCT-SPC) 

Let the N-point DCT of the N-point real signal gj(n) be Gjc(k). The DCT-II is defined as 



Qc(fc) = V/kL giW cos -^ — - — '- 



where 



N ",£,* ' \ N ) 



1/V2, k = 
1, fc^O 



(5) 



(6) 



Gjc(k) is expressed as the absolute value, \Gjc{k) |, and the sign, Ca{k), i.e., 

G ic (k) = \G ic (k)\a Gi (k). (7) 

The DCT sign product is given as 

R a (k)=a Gl (k).a Gl (k), fc = 0,l,---,N-l, (8) 

where cr G .(k) is the sign of Gjc(k). If Gjc(k) is zero, c G .(fc) is replaced by zero. DCT-SPC is 
defined in (18) as 

1 N-l / 7rn1c\ 

r <rM = Jj EWW cos (_)' „ = 0,V,N-1 (9) 
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(a) non-scrambled image 
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(d) scrambled phase-only image 
g<t>i(ni : n 2 ) ^(ni,n 2 ) 

Fig. 3. Images derived from a non-scrambled image and their relationships: The non- 
scrambled image is composed of the magnitude | G z -(/ci, /c^) | and the phase factor ^q^I/^)/ 
while the phase-scrambled image is composed of the magnitude | Q {k\, A^) |, which is identical 
to \Gi(k\,k2)\, and the scrambled phase factor <pQ. {k\, ^2). 



where K^ is the weight, which is generally given as 

K k = (Q) 2 . 



(10) 



The integer displacement value is estimated using (9). The advantages of DCT-SPC over POC 
are computational complexity and memory complexity, because the DCT-SPC uses only the 
DCT signs to estimate translation between signals. The translation with non-integer numbers 
can be estimated by DCT-SPC with fitting function as well as POC (25). 

4. Phase-scrambled signals and matching using POC 

4.1 Template-generation and matching algorithms for secure data management 

Figure 2 shows a model of the template-generation and matching algorithms. In the template- 
generation algorithms, either the DFT or the DCT coefficients of an input image are calculated 
and multiplied by the synchronized signs in order to generate the phase-scrambled coeffi- 
cients. Either the phase-scrambled coefficients or the scrambled phase information, which 
is extracted from the phase-scrambled coefficients, is stored as a template in a database. In 
the secure matching algorithms, when a query image, which is not scrambled, is input, the 
DFT or the DCT coefficients of the query image are calculated and multiplied by the synchro- 
nized signs to generate the phase-scrambled coefficients. The matcher executes phase-based 
correlation between the phase-scrambled coefficients and templates. 

In this section, we explain scrambled signals for POC and an image-matching method using 
POC. We demonstrate that scrambling has no effect on the accuracy of matching. 
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4.2 Phase-scrambled signals 

Let us first consider scrambling of the N-point signal gi(n) for POC. First, N-point signs, 
Sea (k), are generated in random order by a random number generator with a key, oi{, i.e., 

scc { (k)e {1,-1} (11) 

k = 0,l,-' ,N-1. (12) 

In this chapter, the key corresponds to a seed that initializes the random number generator. 
Multiplying the DFT coefficients, G z - (k) , of gi(n) by the N-point signs, s^ (k), yields the scram- 
bled DFT coefficients G/(fc); i.e., 

G J -(fc) = G J -(fc).s«,(fc) (13) 

= G i (k)-e-i^- 1 ^ /2 

= \Gi(k)\e^e-J^i^-^ n/2 (14) 

where s a . (A:) = e~^ Soc ^ )~ 1 ) 7T/2 . G/(fc) is expressed in polar form as 

Q(fc) = |G J -(fc)|^. (15) 

Comparing (14) and (15) yields the relationship between the non-scrambled coefficients and 
the phase-scrambled coefficients: 

\Gi(k)\ = \Gi(k)\ (16) 

and 

cT _ J ik -\-n, Soaik) = -1 ( . 

eik -\e ik , SoCi (k) = i • (17) 

We can conclude that scrambling has no effect on the magnitude. Therefore, the phase- 
scrambled coefficients Gj(k) are expressed in terms of the DFT magnitude, |G z (/c)|, of the 
original signal and the scrambled phase factor, cpQ. (k), as 

Gi(k) = \Gi(k)\^(k). (18) 

The phase-scrambled signal gi(n) is the inverse transform of the phase-scrambled coefficients: 

N-l 
k=0 



?M = ^L Gi(k)W^ k . (19) 



Let us consider the two-dimensional version of the signals as images. Figure 3 shows the 
limited images derived from a non-scrambled image gi (n\, n^), the DFT coefficients of which 
are G{{k\,k'i) = |G z -(/ci ,^2)|^Gj(^l/ ^2)- The phase-scrambled image giin^U'i) protects the 
visual information of the non-scrambled image, as shown in Fig. 3 (b). The DFT magnitude 
of the non-scrambled image and that of the phase-scrambled image are identical. The phase 
factor of the non-scrambled image and that of the phase-scrambled image are different. The 
phase-only image g§ x ,(n\, wi), as shown in Fig. 3 (c), which is derived from the phase factors of 
the non-scrambled image, exposes information about the non-scrambled image, whereas the 
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scrambled phase-only image gi{n\,ri2), as shown in Fig. 3 (d), protects the visual information 

about the non-scrambled image. 

These limited images are expressed as follows: 



g<p i (n 1/ n 2 ) = 

i N-1N-1 

4ee fefe-yw/'w/i (20) 

iV k 1 =0k 2 =0 

gi(n lf n 2 ) = 

1 N-1N-1 

-L £ £ |G z (^,^)|^(^i^2)W N ^ fcl W N ^. (21) 

iN/ fc 1= 0fc 2 =0 



1 



N-1N-1 



^EE fefe^2)W- Wl/Cl W-^ 2 . (22) 

iN/ fc 1= 0fc 2 =0 

where <pQ. (^1,^2) denotes the scrambled-phase factor. 

The phase-scrambled signal and the phase-scrambled coefficients are the space domain rep- 
resentation and the frequency domain representation, respectively. In the following sections, 
we generally do not distinguish the phase-scrambled signal from the phase-scrambled coef- 
ficients, except where confusion may occur. We refer to the phase-scrambled coefficients as 
the phase-scrambled signal for one-dimensional expression or the phase-scrambled image for 
two-dimensional expression. 

4.3 Matching using POC between phase-scrambled signals 

From (3), the normalized cross spectrum, R(p(k), between cpQ 1 (k) and (pQ 2 (k) is given as 

= M*).* Gl (*). S : 2 (*).*£ 2 (*) ( 23 ) 

where, if the same key is used, i.e., s ai (k) = s^ (k), for any k, then 

s CCl (k).sl 2 (k) = s ai (k).s CC2 (k) = l (24) 

and 

R$(k) = R (p (k). (25) 

If the keys are different, i.e., oci ^ #2, then R(p(k) 7^ R<p(k). We conclude that the normalized 
cross spectrum of phase-scrambled signals and that of non-scrambled signals are identical if 
the key is the same, and we can therefore obtain the estimation values with the same accuracy. 

4.4 Scrambling and image matching steps for POC 
4.4.1 Scrambling 

Scrambling proceeds as follows: 

Step 1 The DFT coefficients are calculated from an image. 

Step 2 The signs s ai (k) are generated by a key ocj. 

Step 3 The DFT coefficients are multiplied by the signs according to (13). 
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(a) non-scrambled image 




(c) sign-only image 



(b) sign phase-scrambled 
image 



sign phase- 
scrambled 

DCT 
coefficients 




IGaihM^GAkhh) 9a(ni,n 2 ) 



(d) scrambled sign-only image 



g<T i (ni,n 2 ) 



ga i (n 1 ,n 2 ) 

Fig. 4. Images derived from a non-scrambled image and their DCT relationships: The non- 
scrambled image is composed of the absolute value \Gc(ki,k2)\ of DCT coefficients and the 
DCT signs crQ.(ki,k2), while the sign phase-scrambled image is composed of the absolute 
value \Ga(ki,k2) |, which is identical to |Gq- (^1,^2) |, and the scrambled DCT signs oq. . (/ci,^). 



4.4.2 Image matching for translation 

Image matching using POC for estimating translation between a query and a template is ac- 
complished according to the following steps: 

Step 1 The query is scrambled by the signs that are used for scrambling of the template. 

Step 2 The DFT phase factors are extracted. 

Step 3 The normalized cross spectrum is calculated using (3). 

Step 4 The inverse DFT is applied to the result of Step 3 using (4). 

Scrambling has no effect on the accuracy of image matching, because the effect of scrambling 
is canceled when the normalized cross spectrum is calculated. 

4.4.3 Image matching for rotation and scaling 

The DFT magnitude is used for the estimation of the rotated and scaled values between images 
(6). The phase scrambled method does not distort the DFT magnitude. Therefore, the DFT 
magnitude can be directly used for estimation. The steps for phase-scrambled images are the 
same as those for non-scrambled images. 

5. Sign phase-scrambled signals and DCT-SPC 

In this section, we explain sign phase-scrambled signals and their matching for DCT-SPC. 
The DCT signs express the phases of signals in the transform domain (18). We show that 
scrambling has no effect on the accuracy of image matching. 
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5.1 Sign phase-scrambled signal 

Let us consider sign phase scrambling of gi(n) for DCT-SPC. Multiplying s^ik) in (17) by 
Ga(k) yields the sign phase-scrambled DCT coefficients, Ga(k), i.e., 

G5(fc) = G Ci (k) • s Ui (k) 

= \G Ci (k)\a Gt (k) • Scii (k) (26) 

= \G^i(k)\^(k) (27) 

where the quantities |Gq(/c)| and ^.(k) are the absolute value and sign, respectively. Com- 
bining (26) and (27) gives the quantitative relationships between the sign phase-scrambled 
DCT coefficients, Gq(/c), and the non-scrambled DCT coefficients, Gq(X): 

|G5(*)| = |G Cl -(*)| (28) 

and 

^W=^G,(fc)-s«,(fc). (29) 

Therefore, the sign phase-scrambled DCT coefficient, Ga(k), is expressed in terms of the ab- 
solute value, | Gci(k) |, of the non-scrambled signal and the scrambled DCT sign, Oq. (k), as 

G5(fc) = |G G (fc)|^(fc) (30) 

The sign phase-scrambled signal is the inverse transform of the sign phase-scrambled DCT 
coefficients, i.e., 

htn-i ( nin + -)k\ 

fcW = V n ? C i|G Q (fc)l^(fc)cos ( l ^ 2 M • (3D 

The sign phase-scrambled signals are real numbers. The DCT-SPC uses the DCT signs, which 
are extracted from the transform of the sign phase-scrambled signals. 

Figure 4 shows the images derived from a non-scrambled image, as shown in Fig. 4 (a), and 
the relationships between these images. As shown in Fig. 4 (b), the sign phase-scrambled 
image protects the information about the non-scrambled image. The DCT signs of the non- 
scrambled image and that of the sign phase-scrambled image are different. The sign-only 
image, as shown in Fig. 4 (c), which is derived from the DCT signs of the non-scrambled 
image, exposes the information of the non-scrambled image, while the scrambled sign-only 
image, as shown in Fig. 4 (d), protects the information about the non-scrambled image. 

5.2 Matching using DCT-SPC between sign-scrambled signals 

In the case of the sign phase-scrambled DCT coefficients, the DCT sign product is also invari- 
ant if the same key is used. From (8), the DCT sign product, R a {k), between o^{k) and &G 2 {k) 
is given as 

Mfc)=5£(*)-5£(*). (32) 

From (8), 

RAk) = a Gl (k)s ai (k)-a Gl (k)s ai (k). (33) 
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If the same key is used, we obtain the same result under scrambling, i.e., 

%(k) = R a (k). (34) 

If the keys are different, the result is R a (k) ^ R(r(k). This can help prevent illegal use of the 

image matching. 

We can thus conclude that there is no effect of scrambling on registration accuracy. 

5.3 Scrambling and image matching steps for DCT-SPC 

5.3.1 Sign phase scrambling 

Scrambling proceeds as follows: 

Step 1 The DCT coefficients are calculated from an image. 

Step 2 The signs s ai (k) are generated by a key &[. 

Step 3 The DCT coefficients are multiplied by the signs s a . (k) according to (26). 

5.3.2 Image matching for translation 

Image matching using DCT-SPC for estimating translation between a query and a template is 
accomplished according to the following steps: 

Step 1 The query is scrambled by the signs that are used for scrambling of the template. 

Step 2 The DCT signs are extracted. 

Step 3 The DCT sign product is calculated using (8). 

Step 4 The inverse transform is applied to the result of Step 3 using (9). 

Scrambling does not affect the accuracy of image matching, because the effect of scrambling 
is canceled when the DCT sign product is calculated. 

6. SIMULATION 

6.1 Translation (synchronized phase scrambling) 

Translation estimation experiments were performed using non-scrambled images and syn- 
chronized phase-scrambled images. Figure 5 shows the test images: (a) is a 256 x 256 non- 
scrambled image, (b) is the shifted image of (a) by 20 pixels in both the horizontal and vertical 
directions, and (c) and (d) are the phase-scrambled images of (a) and (b), respectively. Note 
that the phase-scrambled images were generated with the same key, which initializes the ran- 
dom number generator, i.e., oc\ = #2, s ai (k) = s a2 (k), for any k. We refer to the use of the same 
key as synchronized phase scrambling. We executed POC between the two non-scrambled 
images, (a) and (b), and POC between the two phase-scrambled images, (c) and (d), as shown 
in Fig. 5. Figure 6 shows that the POC surface between phase-scrambled images in (21) was 
the same as the POC surface between non-scrambled images in the case of synchronized phase 
scrambling. Phase scrambling has no effect on the accuracy of image matching. 
We also performed DCT-SPC between the two non-scrambled images and DCT-SPC between 
the corresponding sign phase-scrambled images. Note that the sign phase-scrambled images 
were generated using the same key. Figure 7 shows the DCT-SPC surface between sign phase- 
scrambled images and that between non-scrambled images. Sign phase scrambling was con- 
firmed to have no effect on the accuracy of image matching. 
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We also confirmed the estimation of translation with subpixel accuracy. The same accuracy 
was obtained for phase-scrambled images as that for non-scrambled images. (Details are omit- 
ted due to space limitations.) 





(c) phase-scrambled (d) phase-scrambled 
image of (a) image of (b) 

Fig. 5. Test images (256 x 256): (a) is the original image, and (b) is the shifted image of (a), (c) 
and (d) are the phase-scrambled images of (a) and (b), respectively. 
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o o 

(a) non-scrambled image 



r4>(ni,n 2 ) 




(b) scrambled image with the same key, oc\ = a: 2 
Fig. 6. Estimation of translation using POC: (a) is the POC surface between the non-scrambled 
images, and (b) is the POC surface between the phase-scrambled images, (a) and (b) are 
identical, and an acute peak appears at the location expressing the translational displacement. 



Phase Scrambling for Image Matching in the Scrambled Domain 



409 




(a) non-scrambled image 




(b) scrambled image with the same key, oc\ = &2 
Fig. 7. Estimation of translation using DCT-SPC: (a) is the DCT-SPC surface between the non- 
scrambled images, and (b) is the DCT-SPC surface between the sign phase-scrambled images, 
(a) and (b) are identical, and an acute peak appears at the location expressing the translational 
displacement. 



6.2 Effect of noise on image matching (synchronized phase scrambling) 

Figure 8 shows the effect of noise on image matching. The test image is shown in Fig. 5 (a), 
and the shifted image is shown in Fig. 5 (b). First, the noise, which consisted of Gaussian 
random numbers with zero mean and a standard deviation of 25, was added to the shifted 
image in the space domain. Figure 8 (a) shows the POC surface between the image and the 
shifted image with noise. Compared with Fig. 6 (a), the effect of noise on the POC surface is 
clear. 

Next, we scrambled these two images and performed POC. Figure 8 (b) shows the results. We 
confirmed that (a) and (b) in Fig. 8 were identical. We can conclude that scrambling has no 
effect on image matching. 
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(a) POC between the non-scrambled image and 
the non-scrambled shifted image with noise 




(b) POC between the phase-scrambled image and 
the phase-scrambled shifted image with noise 
Fig. 8. POC with noise: Gaussian random number with zero mean and a standard deviation 
of 25. (a) and (b) are identical. 



6.3 Translation (desynchronized phase scrambling) 

Translation estimation experiments were performed between the desynchronized phase- 
scrambled images and desynchronized sign phase-scrambled images. That is, the phase- 
scrambled images were generated with different keys, i.e., oc\ ^ oli> The sign phase-scrambled 
images were also generated with different keys, i.e., oc\ ^ oli> Figure 9 shows both the 
POC surface between phase-scrambled images and the DCT-SPC surface between sign phase- 
scrambled images. A distinct peak expressing the translational displacement did not appear 
on either the POC surface and the DCT-SPC surface. The properties of phase-scrambled im- 
ages with different keys provides a countermeasure against cross-referencing. 
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(a) POC with different key ol\^ oli 




(b) DCT-SPC with different key oc x ^ oc 2 
Fig. 9. Estimation of translation with different key, oc\ ^ &2 (a) is the POC surface between 
phase-scrambled images with different keys, (b) is the DCT-SPC surface between sign phase- 
scrambled images with different keys. A distinct peak expressing the translational displace- 
ment did not appear in (a) or (b). 



6.4 Rotation and Scaling 

Rotated and scaled values are generally estimated using the DFT magnitude. This is based 
on the fact that the DFT magnitude contains information related to the rotated and scaled 
values, which are independent of translation. Note that the log-polar transform is applied to 
the DFT magnitude in order to reduce the rotated and scaled values to vertical and horizontal 
translations, respectively (6). 

The rotated and scaled values between two images, as shown in Fig. 10, were estimated under 
two conditions: non-scrambling and phase scrambling with the same key. We confirmed that 
the POC surface between phase-scrambled images with the same key and that between non- 
scrambled images were identical. This result is trivial, however, because the presented phase 
scrambling does not distort the DFT magnitude, as shown in (16). 

The phase scrambling does not affect the image matching, and we can perform image match- 
ing without descrambling. 
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(a) original image 



(c) scaled image, 
scale factor: 1.2 



(b) rotated image, 
angle: 15°. 

Fig. 10. Estimation of the rotated and scaled values. Log-polar transform is used to reduce 
rotated and scaled values to translational values (6). 



7. Conclusion 

The presented scrambling enables image matching using invisible images. In addition, im- 
age matching between phase-scrambled images is performed with the same accuracy as that 
between non-scrambled images. We have explained how to generate phase-scrambled sig- 
nals that protect the information in the original image and demonstrated that the presented 
synchronized phase-scrambling maintains the relative relationship between two signals math- 
ematically. We have shown that the presented scrambling is applicable to both DFT and DCT 
coefficients, and therefore secure image matching for phase-based correlation is achieved. 
In particular, DCT-SPC is closely related to the image compression method. Application of 
the presented scrambling in areas such as image communications and image compression 
appears promising. 

In the presented scrambling, the management of keys depends on desired systems. Mean- 
while, for visual protection, there is one-time key based phase scrambling which does not 
require the management of keys (26; 27). 
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1. Introduction 

A significant number of the algorithms that are currently employed in commercially available 
single-channel speech enhancement products are waveform filtering based methods. Waveform 
filtering implies that an appropriately chosen filter 1 is applied to the incoming noisy speech 
data in order to change or smooth the shape the resulting filtered waveform. The objective 
of the filtering is usually to either improve the perceptual quality of the output -or- to improve 
the recognition rate of a subsequently used speech recognition system. Prominent examples 
of waveform processing are the Wiener filtering extensions proposed by McAulay & Malpass 
(1980) and Ephraim & Malah (1984). Other examples include schemes that employ wavelets 
by Hu & Loizou (2004) and modifications of the iterative Wiener filter and the Kalman filter by 
Mouchtaris et al. (2007). Also related are the spectral subtraction method developed by Boll 
in 1979 (see the text by Deller et al. (1993)) and its powerful extension, the multiband spectral 
subtraction described in the text by Loizou (2007). 

The success of many waveform filtering based methods is due to their relative (computational) 
simplicity and robustness. A disadvantage of filtering based methods, however, is that they 
are never able to completely remove the noise. They are usually aiming to achieve a reasonable 
tradeoff between a desired reduction of the noise and an undesired but inadvertent distortion 
of the targeted signal. 

The search for a denoising paradigm that, at least in theory, allows for potentially "perfect" 
enhancement has motivated many researchers to study model based denoising methods. In 
model based denoising a parametric model for a speech signal (which may be determinis- 
tic or stochastic in nature) is used instead of a general waveform model. A popular choice 



1 The employed filters are typically linear but potentially time-variant. 
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for a speech model in this context is the harmonic plus noise model (HNM) which was studied 
amongst others by Zavarehei et al. (2007). Related is also the work by Zhao & Kleijn (2007) 
on the modelling and estimation of speech and noise gains via hidden Markov models. Code- 
books of linear predictive coefficients and their employment for speech denoising within a 
maximum-likelihood framework were studied by Srinivasan et al. (2006). A minimum mean 
square error approach for denoising that relies on a combined stochastic and deterministic 
speech model was studied by Hendriks et al. (2007). 

The model based speech denoising method discussed in this chapter was proposed by Xiao 
et al. (Aug. 2008) and (Apr. 2009). It is inspired by the increasing success of inventory based 
speech synthesis systems as discussed in a review paper by O'Shaughnessy (2007). In this work it 
is assumed that speaker enrollment and noise enrollment are feasible. The speaker enrollment 
procedure provides training data that can be appropriately clustered and used as an inventory 
for a "clean" speech signal model. The inventory based denoising is supported by a statistical 
analysis of the speech signal under clean and noisy conditions. 

One of the disadvantages of this method in comparison to other model based approaches is 
its high computational complexity. The procedure requires, in its originally proposed form, 
a very large number of floating point multiplications. The bottleneck of the procedure can 
be found in correlation operations that need to be carried out over large data records. In this 
chapter we are describing a modification of the original approach that incorporates a. fast al- 
gorithm for the denoising stage. The proposed method substantially reduces the amount of 
necessary multiplications via the employment of a set of number theoretic transforms (NTTs, 
Blahut (1987)). Number theoretic transforms allow the computation of correlations in fixed- 
point arithmetic with a substantial reduction in multiplications. However, NTTs come gener- 
ally at the expense of reduced computational accuracy due to the required signal quantization. 
We present an approach that balances a dramatic reduction in computational complexity with 
only a very slight reduction in perceptual denoising performance. 

The chapter is divided into two main sections. Section 2 provides a summary of the proposed 
denoising paradigm. A full and detailed description of the method is beyond the scope of this 
chapter. The interested reader will find it in the two papers by Xiao et al. (Aug. 2008) and 
(Apr. 2009). Section 3 focuses explicitly on the parts of the procedure that can be improved 
with a fast algorithm. 

2. The Denoising Paradigm 

The method proposed by Xiao et al. (Apr. 2009) can be divided into three main tasks: (i) a 
system training task, (ii) the signal preprocessing task, and (iii) the signal denoising task. The 
main contribution of this work can be found in the modification of the signal denoising task. 
For the reader's convenience, however, we are providing a cursory overview of the entire 
system in this section. The description follows closely in structure and notation with that of 
the original paper by Xiao et al. (Apr. 2009). Many key details, however, are omitted here. The 
interested reader may want to consult the original paper for a comprehensive presentation. 
The system training task consists of the development of a speech waveform inventory, two mel- 
frequency cepstral coefficient (MFCC) codebooks (under clean and noisy conditions), and a hid- 
den Markov model (HMM). The HMM is used to model the codeword transition statistics under 
clean and noisy conditions. The system training task is discussed in some greater detail in sec- 
tion 2.1. 

The procedures of the signal preprocessing task are adjusted according to the expected noise 
type. Three different noise types are considered in the original paper. They are white 
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noise, colored noise, and non-stationary noise. No preprocessing is performed in the case 
of white noise. Stationary colored noise requires preprocessing with a prewhitening filter. Non- 
stationary noise is preprocessed with a combination of a short-time power spectral estimator 
(via harmonic tunnelling, Ealey et al. (2001)) and subsequent Wiener filtering. 
Lastly, the speech denoising task combines the results of the preprocessing with the results of 
a state sequence computation from the trained HMM as described in section 2.1. Suitable sections 
from the speech inventory are chosen through an inventory unit selection scheme and are then 
concatenated to form the targeted denoised speech signal (see section 2.2). The inventory 
unit selection scheme constitutes the computational bottleneck of the procedure. Most other 
components of the method have a computational complexity that is comparable to that of 
other model based methods. The complexity of the inventory unit selection scheme, however, 
dominates the overall processing requirement by an order of magnitude. The fast processing 
algorithm presented in this work focuses therefore exclusively on the inventory unit selection. 
It is comprehensively described in section 3. 

Throughout this chapter we are using a mathematical notation that is consistent with the one 
introduced in the paper by Xiao et al. (Apr. 2009). At the denoising stage we assume that we 
observe a signal x[n] which consists of speech s[n] that is uttered by the enrolled speaker and 
is distorted by zero mean additive noise v[n], i.e. x[n] = s[n] + v[n\. At the training stage we 
use s[n] to, similarly, denote the speaker enrollment data. System training is done off-line from 
speaker-specific pre-recorded clean training signals. For simplicity we assume that all training 
records of speech are concatenated into one long training sequence s[n]. 
An accurate description of the enhancement procedure requires the definition of speech units 
or frames. We represent a unit as a vector of N successive samples of a signal: 

s n = [ s[n - L] s[n - L + 1] ... s[n - L + N - 1] ] T . (1) 

Note that in section 3 we employ signal segments of a different length, i.e. segments with a 
processing block-length of K (with K > N, see equations (10) and (11)). The amount of overlap 
between adjacent frames is controlled by a step size L. If i denotes a unit (or frame) index then 
the associated vector is written as s z x- Symbols x n , v n , and s n are defined analogously to 
equation (1). Symbol S is used to denote our speech-waveform-unit inventory. Set S consists of 
all clean training data frames s n (Vn, i.e. with a step size of one) with the exception of data 
frames that are entirely silent. Data frames are considered entirely silent if the total frame 
energy falls below a certain minimal level. 

The fundamental paradigm behind the considered denoising method is quite simple: find a 
mapping x zL — >• s n n) that associates a specific inventory frame s n n\ to every observed noisy 
frame x z x- The complexity of the method arises from the fact that this mapping is generally not 
fixed, but time-variant and context dependent. A resulting denoised signal s[n] is obtained by 
"concatenating" the found frames s n n\ via a sinusoidal model based resynthesis technique. The 
employed resynthesis technique is similar to the one described in the text by Quatieri (2002). 
Please refer to the original paper by Xiao et al. (Apr. 2009) for the details. 

2.1. System Training and State Sequence Estimations 

The system training stage is used to achieve two separate goals: (1) to provide the denoising 
procedure with an inventory of available speech units and (2) to generate a hidden Markov model 
that describes transition statistics within the inventory. An illustration of the inventory design 
procedure is shown in figure 1. All inventory elements s n that belong to a similar phonemic 
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Fig. 1. An illustration of the data clustering method used by the denoising procedure by Xiao 
et al. (Apr. 2009). Training data is segmented, MFCC coefficients are extracted, and frames 
with "similar" coefficient vectors are lumped into one of M = 50 inventory clusters. 



function 2 are grouped into the same cluster. The purpose of the grouping is to be able to 
study the statistical properties of the group as a whole and then apply a resulting statistical 
description in the denoising process. The procedure requires the construction of two mel- 
frequency cepstral coefficient (MFCC) codebooks: a clean MFCC codebook C = \C\, Ci, . . . , Cm} 
and a noisy MFCC codebook C = { C\, Ci, • . • , ^m}- The codebooks contain the average of 
the MFCC vectors of all clean /noisy inventory units within a respective cluster. The details 
of the clustering procedure and the inventory design are omitted here. Please consult the 
paper by Xiao et al. (Aug. 2008) for a comprehensive description. An illustration of the noisy 
codebook design procedure is shown in figure 2. To maintain compatibility with the notation 
introduced in Xiao et al. (Aug. 2008) we will refer to the resulting cluster sets of inventory 
vectors s n with K k for k = 1, 2, . . . M. 

Given the clean and noisy MFCC codebook vectors for each inventory cluster it becomes pos- 
sible to estimate the cluster transition statistics for the given speaker. An illustration of the 
considered statistical description after Xiao et al. (Aug. 2008) is shown in figure 3. We use 
s n —> k to indicate the cluster membership of inventory frame s with cluster k. Similarly, we 
define x.n — >• j to indicate that the incoming noisy frame xn is vector quantized via the noisy 
codebook C into cluster j. With a simple counting process we can estimate the first-order 
temporal state transition probabilities, i.e. 



l Xj 



Probf 



5 (i+l)L -»■ 1 I s iL 



(2) 



Similarly, we can convert our sequence of noisy training frames x zL into an observation code 
sequence. Again, with a counting process we can estimate the noise induced observation proba- 
bilities jointly from our clean and noisy training data: 



Q k/j = Frob[x iL ->j\$ iL ->k], 



(3) 



2 We are using the term phonemic function in reference to a general, function carrying unit of a language. 
The group may or may not match with an actual phoneme defined for that language. 
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Fig. 2. An illustration of the generation of the noisy MFCC codebook as proposed by Xiao 
et al. in (Aug. 2008) and (Apr. 2009). Training noise is added to the elements of the clean 
inventory. The noisy MFCC codebook arises from the average of the MFCC vectors computed 
from the respective distorted signals within each clean cluster. 
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Fig. 3. An illustration of the statistical description of the considered cluster membership after 
Xiao et al. (Aug. 2008). The observation of a "noisy" code is statistically related to the "true" 
cluster membership of the underlying clean signal segment. 



The transition probabilities P^y and Qy are both used in the denoising process. The statis- 
tical description enables us to define an "optimal" sequence k pt{i) of cluster memberships 
for incoming testing frames x zL - The sequence is optimal in the sense that the "most likely" 
inventory element s n n\ to represent the denoised frame for x zL is found in set IK^ ny Again, 
the details of how to find the sequences k op t(i) are omitted. A comprehensive description is 
found in the paper by Xiao et al. (Aug. 2008). 
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2.2. Speech Denoising 

After completion of the described system training we can begin to denoise new incoming 
signals x[n]. The denoising procedure can be broken down into 4 separate steps: (1) the es- 
timation of the "optimal" cluster membership sequence k pt(i) (as discussed in the previous 
section), (2) the preprocessing, i.e. prewhitening, of the noisy signal x[n], (3) the identification 
of the best match for each xn in K fco n\, i.e. the intra cluster frame matching, and (4) the "con- 
catenation" of the resulting inventory frames to resynthesize the targeted denoised signal. 
As indicated in the previous section, we will omit the details of the k pt(i) -sequence estima- 
tion. Sequence k pt(i) can be computed fast and efficiently via the Viterbi algorithm, as de- 
scribed in the paper by Xiao et al. (Aug. 2008). We also omit most of the details of the data 
preprocessing, as they have been comprehensively described in the same paper. We will, how- 
ever, briefly describe the underlying principles of the intra cluster frame matching since it is the 
target of the proposed fast algorithm described in section 3. 

In a first step we define a similarity measure between a noisy frame Xj L and an inventory 
element s n . The choice of the similarity measure proposed in the paper by Xiao et al. (Apr. 
2009) was guided by fundamental detection theory. If we are assuming a maximum likelihood 
criterion and if the additive noise v zL is independent white Gaussian noise then a correlation 
detector should be used (see Poor (1994)). Since the power of the training frame and the 
testing frame may be significantly different a power normalization was proposed as well. The 
resulting similarity measure becomes 

T * 

0-(x iL ,S n ) = == / (4) 

V||x;lII ^ 2 ' ll s "ll 



in which \/||x z x|| 2 — V 2 represents the estimated power of the underlying clean speech s[n]. 
If Vfi contains colored noise then a prewhitening filter is used before the correlation detector. 
With h^ denoting the impulse response of the prewhitening filter we obtain 

, , fl x (x iL *h w ) T (s n *h w ) 

a{x iL/ s n ) = , , (5) 

:,7. *h r J| 2 - y 2 • llstr *h 7 J| 



where we use V 2 = E{(v n * h^) T (v n * h^)} to denote the variance of the prewhitened noise. 

A discussion of the non-stationary noise case is omitted here since it is effectively using the 

same similarity measure as the colored noise case. 

After the generation of an appropriate similarity measure between an incoming noisy frame 

Xj L (or Xji) and an inventory element s n we can define an optimal intra cluster match s( l,k ) 

via 

g(U0 _ ar g max o~(xj L/ s n ). (6) 

s„eK fc 
In a last step we need to resynthesize our targeted signal. First, we are replacing each frame 
Xji with the inventory frame §(^pt(0) / i.e. Xj L — ^ §( z / fc opH z )) / anc [ second, we reconcatenate the 
resulting frames via a sinusoidal model expansion similar to the one proposed by Quatieri (2002). 
The reconcatenation with the sinusoidal model is important to minimize phase incompatibil- 
ities at the frame boundaries. 

The performance of the proposed method was evaluated by Xiao et al. (Apr. 2009) with exper- 
iments over a subset of the CMU_ARCTIC database from the Language Technologies Institute 
at Carnegie Mellon University 3 . Data processing was conducted at a sampling rate of 8 kHz 



' The corpus is available at <http://www.festvox.org/cmu_arctic>. 
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and with a segment length of N = 160 samples and a step size of L = 80 samples. The 
targeted signal-to-noise ratio was 10 dB. The quality of the resulting denoised speech was as- 
sessed with the Perceptual Evaluation of Speech Quality 4 (PESQ) measure. For white noise Xiao 
et al. reported an improvement of up to 1.06 points on the PESQ scale. For colored noise an 
improvement of up to 0.87 points was reported. The performance of the presented method 
compared favorably with other state-of-the-art denoising methods. 

3. Fast Processing Methods 

As mentioned earlier, the bottleneck of the proposed denoising procedure, in terms of compu- 
tational complexity, can be found in the maximization of equations (4) and (5) as expressed in 
equation (6). In the experiments conducted by Xiao et al. (Apr. 2009) training sets of around 
1 hour in length were used. If we operate at a sampling rate of 8 kHz and with a number of 
M = 50 clusters then we can expect to have around 500 • 10 3 to 600 • 10 3 samples per cluster. 
For the denoising of each incoming frame we need to correlate a vector of length 160 with 
the entire data set contained in the cluster targeted by k op t(i). The resulting computational 
complexity per frame is therefore huge, if no fast computational procedures are involved. 
For the remainder of this section we will discuss methods that can dramatically reduce the 
computational complexity of the maximization implied in equation (6). In a first step we are 
moving to a slightly simplified similarity measure since for a fixed x zL the terms vTi^L II 2 — ^ 2 

and y\\xii * h w \\ 2 — V& remain unchanged and can therefore be dropped from the computa- 
tion: 

T ~ 

<7-(x,s„) = -rnr-iT' ( 7 ) 

||Sn|| 

Similarity measure (5) can be modified accordingly. The respective result for equation (5) is 

obtained by substituting x zL * h w and s n * h w into equation (7). 

The fast computation of (7) for all frames in a given cluster (as expressed in equation (6)) is 

accomplished in three steps: 

1. Quantization of the elements in x and s n . 

2. Computation of x T - s n with an overlap-add based convolution procedure via number 
theoretic transforms (NTTs). 

3. Recursive computation of \\s n \\ and a(x, s n ). 

We begin by applying a uniform scalar quantizer (see Sayood (1996)) to all elements of our 
(possibly preprocessed) incoming frame x and the elements of our inventory vectors s n . The 
quantizer is designed to assign a unique integer between — / and +/ (/ G N) to every value 
in x and s n . An optimal choice of / is dependent on three things: (1) the employed frame 
length N, (2) the statistics of our training data, and (3) the parameters of the employed number 
theoretic transforms. Good choices for / are discussed in section 3.3. 

For simplicity of notation we assume for the remainder of this section that symbols x and s n 
and the associated signals x[n] and s[n] are quantized versions of the original signals, i.e. all 
elements of these signals and vectors are integers in the range —/... + /. It is important to em- 
phasize that we are not operating with this kind of quantized data in other components of the 
proposed method, especially in the step x zL — >• s^' WW) during the target signal resynthesis 
where we want to use the original inventory data and not the quantized one. 



4 The PESQ measure, an ITU recommendation, is aiming to asses the subjective quality of speech. Please 
refer to the text by Loizou (2007) for the details. 
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3.1. Preliminary Computations and Notation 

Before we delve into the fast computation of x T - s n it is beneficial to first briefly discuss an 
efficient way to compute the term ||s n || in (7). For notational convenience we introduce the 
shifted inventory signal s'[n] = s[n — L + N — 1] and its square g[n] = s f [n] • s'[n\. We can use 
g[n] as an input to the following recursive system with output £[ri\: 

S[n]=Z[n-l]+g[n]-g[n-N\. (8) 

Term ||s n || is then obtained from ||s n || = \/£[tt]. The computation of ||s n || therefore requires 
one multiplication, two additions 5 , and one square root operation (table lookup) per sample. 
A major step in simplifying the computation of x T - s n is obtained from recognizing that the 
inner product in (7) is equivalent to a convolution operation (indicated with symbol *): 

N-l 

* T '$n= Yll x }N-k-s[n-k + N-L-l] = [x] N - k *s'[n]. (9) 

k=0 

We use the notation [x] k to indicate the fc th element of vector x with [x]^ = if k < 1 and k > N. 
The convolution of equation (9) is further broken down by segmenting s' [n] into segments of 
length R. The segments are zero padded to arrive at a processing block-length of K samples: 

s' k =[ s f [kR] s f [kR + l] ... s'[kR + R-l] ... ] T . (10) 

K-R 
Similarly we are defining a time-reversed and zero padded input signal vector x' as: 

x ; = [ [x] N [x] N _! ... [x] 2 [x]x . .. ] T . (11) 

K-N 

The convolution operation can then be performed via number theoretic transforms (NNTs, 
see Blahut (1987)). The details of the proposed NTT operation are described in section 3.2. If 
we assume that we have access to the output vector y' k = NTTConvjx 7 , s^} of the proposed 
X-point NTT convolution of x' and s' k then the inner product in equation (9) becomes: 

X T 'S„ = Ek[Yk}n+l-kR- (12) 

The overlap and add method implied in equation (12) requires (K — R) additions for each 
block of length K, plus the operations necessary for NTTConv. 

3.2. Number Theoretic Transforms 

Number theoretic transforms can be used for efficient computations of convolutions if the 
underlying data is, as indicated earlier, discretized or quantized (see Blahut (1987)). NTTs 
generally operate in finite fields or Galois fields. The order p of the field is typically a prime 
number, in which case all operations within the field (addition, subtraction, multiplication, 
and division) are executed via a modulo-p arithmetic (see Blahut (1987)). 
Not all number theoretic transforms are necessarily well suited for the development of fast 
algorithms. NTTs of a certain subclass, known as Fermat NTTs, however, have properties that 
make them superior to the commonly used fast Fourier transform (FFT, see Proakis & Manolakis 



5 We are counting subtractions and additions as the same since they share roughly the same computa- 
tional complexity. 
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(1996)) in computing convolutions within a fixed-point arithmetic. The advantage of such 
NTTs are that: (1) an NTT can be implemented in real-valued arithmetic (i.e. it does not re- 
quire an underlying complex number representation), and (2) many of the multiplications re- 
quired for the computation simplify to shift operations if the underlying processing hardware 
is utilizing binary number representations. 

NTTs have, however, three important limitations that render their practical implementation 
significantly less flexible than that of the FFT: (1) the processing block length K is tied to (i.e. 
not independent of) the order p of the underlying number representation, (2) NTTs only exist 
for a very limited number of combinations of K and p, and (3) internal overflow errors during 
convolution computations cannot be detected and /or flagged. A general discussion of all of 
these problems is beyond the scope of this book chapter. The interested reader may consult 
the literature, especially the text by Blahut (1987), for a detailed discussion. We will address 
the three issues above only within the context of the proposed denoising scheme. 
Out of the general set of possible combinations for K and p we found that K = 1024 and 
p = 2 16 + 1 (Fermat prime) are quite well suited for the proposed algorithm. We begin by 
considering a general integer vector v = [ v\ V2 . . . v& ] T of length K. More specifically, 
we assume that all elements v k of v are integers between — ^- and + ^- . Furthermore, we 
define the following warping operation: 

warpK}=( Vk ***** (13) 

The notation v = warp{v} refers to an application of the warp-function on vector v on an 
element-by-element basis. We also require the following dewarping mapping: 

dewaip{* fc } = ( ? **'% d4) 

{ v k -p i£v k > V- 

Again, v = dewarp{v} refers to an application of the dewarp-f unction on vector v on an 
element-by-element basis. Given the parameters K = 1024 and p = 2 16 + 1 we arrive at the 
following definition for the employed NTT: 

1023 

V = NTT{v} such that [V] fc+1 = £ cv lk [v] m mod p for k = . . . 1023. (15) 

z=0 

Number to must be chosen such that a; 1024 mod p = 1. There are a number of values to 
that satisfy this conditions. Not all choices, however, are well suited for the design of a fast 
algorithm. It is possible to apply the radix-two Cooley-Tukey divide-and-conquer approach (see 
Blahut (1987)) to equation (15). With the special choice of w = 18990 we obtain: 
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mod p for k f f k" = . . .31. (16) 



It is, therefore, possible to divide the 1024-point NTT from equation (15) into two sets of 32 
sub-NTTs of length 32 each and one set of 1024 multiplications with cv . Furthermore, the 
32-point sub-NTTs can be computed without any multiplications since a multiplication with a 
power-of-two number is equivalent to a shift operation if the underlying processing hardware 
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operates on a binary number system. A detailed analysis of equation (16) reveals that equa- 
tion (15) can be computed with roughly 1024 multiplications, 5120 additions, and 5120 shift 
operations. 

An important aspect of the computations in equations (16) and (15) is the modulo reduction 
of order p. Given the standard approach to modulo reductions one might suspect that each 
reduction comes at the cost of an integer division. In the given case of p = 2 16 + 1, however, 
it becomes possible to reduce the complexity of a modulo reduction to that of an addition, if 
the underlying processing hardware operates on a binary number system. To that end, we 
can group adjacent bits in our underlying number representation into blocks of 16. All blocks 
with bits higher than 16 are shifted down to line up with the least significant bit and then 
added block-by-block with an alternating sign. The details of the implementation are readily 
found in the literature (see also Blahut (1987)). Due to the cyclic nature of the p = 2 16 + 1 
reduction it is possible to build the require modulo operation directly into the hardware of the 
employed multiplier, adder, and shifting units. We, therefore, do not count modulo operations 
separately in our complexity analysis. 

The computation of convolutions via NTTs requires us to also consider the inverse NTT. Sim- 
ilarly to equation (15) we define: 

1023 

v = NTT-^V} such that [v] fc+1 = X" 1 £ co~ ik [V] /+1 mod p 

i=0 

for k = . . . 1023. (17) 

Note that (K _1 ) represents the integer with the property (K _1 ) • K mod p = 1. The inverse 
NTT can also be computed via the radix-two Cooley-Tukey divide-and-conquer approach: 
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31 
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mod p 

for *:',*:" = 0... 31. (18) 



The computational complexity of the NTT and the inverse NTT are therefore the same, except 
we have an additional set of 1024 multiplications with X -1 in the case of the inverse NTT. We 
will see in section 3.3 though that the scaling with X -1 can be omitted when we apply the 
inverse NTT to our proposed fast computation procedure. 

We are now in the position to define the computation of y' k as used in equation (12). The oper- 
ation requires two warping operation, one dewarping operation, two NTTs, and one inverse 
NTT: 

y' k = NTTConv{x',s^} = dewarplNTT-^NTTjwarplx 7 }} ©NTT{warp{s^}}}}, (19) 

in which denotes element-by-element-wise vector multiplication. 

3.3. Complexity Analysis and Perfomance 

A successful implementation of the fast algorithm proposed in the previous two sections re- 
quires a careful definition of our quantization granularity 6 /. If / is too big then we are likely 
to receive too many (undetectable) overflow errors in the NTT based convolution operation. 



' Note that the effective number of quantization levels for our data is given by 2/ + 1. 
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The resulting intra cluster frame matching becomes unreliable and the perceptual quality of 
the proposed denoising method suffers. If we pick / too small then the effective quantization 
granularity of our data becomes too coarse. Again, the resulting intra cluster frame matching 
becomes unreliable and the perceptual quality is reduced. 

In experiments over the same data set that was used in the original performance analysis 
by Xiao et al. (Aug. 2008) we found that the error count in the intra cluster frame matching 
procedure remained relatively unaffected by the number of employed quantization levels if 
the number did not drop significantly below 60, i.e. 2/ + 1 > 60. Similar experiments revealed 
that we receive virtually no overflow errors in our procedure 7 if / < 35. A recommended 
range for / is therefor between 30 and 35. 

To maximize the efficiency of the proposed NTT based convolution it is best to pick K = 
N + R — 1. With a processing block-length K of 1024 and a frame length N of 160 we obtain 
R = 865. 

To obtain reasonably normalized numbers for the computational complexity of different solu- 
tion approaches for equation (6) we decided to reference all operation counts to an equivalent 
count for each inventory sample. A direct, brute force, computation of equation (6) requires 
320 multiplications /sample and 318 additions /sample. We technically also require one divi- 
sion/sample and one square-root-operation/ sample. The division and the square root, how- 
ever, are a part of all considered algorithms and are therefore omitted in the overall counts. 
For comparison we consider the proposed fast convolution approach with a conventional 
radix-two fast Fourier transform (FFT) instead of the proposed NTT. A 1024-point FFT requires 
9216 complex multiplications and 10240 complex additions 8 (see Blahut (1987)). Each com- 
plex multiplication can be evaluated with 3 real multiplications and 5 real additions. We, 
therefore, obtain 27648 real multiplications and 56320 real additions. The disadvantage of a 
complex arithmetic of the FFT is partially alleviated by the fact that we can typically process 
two FFTs with real imputs with a single FFT with complex inputs (see Proakis & Manolakis 
(1996)). Operations are consequently cut in half and we obtain as a final count for a 1024- 
point FFT 13824 (real) multiplications and 28160 (real) additions. The FFT equivalent of equa- 
tion (19) can be evaluated on-line with one 1024-point FFT, one 1024-point inverse FFT and 
1024 complex multiplications. Note that we only need one FFT to compute (19) on-line since 
the corresponding FFTs of our inventory s[n] can be precomputed off-line. Furthermore, we 
do not need to consider the additional scaling factor of ^ in the inverse FFT since the scaling 
becomes immaterial in the subsequent maximum search. Furthermore, we receive an addi- 
tional number oiK — R = N — 1 = 1023 additions due to the overlap and add procedure from 
equation (12). 

In summary, we require 2 x 13824 + 3 x 1024 = 30720 multiplications and 2 x 28160 + 5 x 
1024 + 1023 = 62463 additions to compute the required 1024-point convolution with an FFT 
based approach. The convolution operation has to be repeated every R = 865 samples. On a 
per-sample count we obtain 35.52 multiplications /sample and 72.22 additions /sample. Tech- 
nically we need to also add in the one multiplication/ sample and the two additions /sample 
for the separate computation of \\s n || . 



7 Assuming N = 160, K = 1024, and p = 2 16 + 1. 

8 The complexity analysis presented here may slightly differ from complexity computations from other 
sources. The main differences in computation counts are usually due to differences in how trivial mul- 
tiplications are considered. We decided to include the count of trivial multiplications for the FFT as 
well as the NTT. 
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The computation of a 1024-point NTT after section 3.2 requires 1024 multiplications, 5120 
additions, and 5120 shift operations. Similarly to the computations for the FFT we re- 
quire 2 x 1024 + 1024 = 3072 multiplications, 2 x 5120 + 1023 = 11263 additions, and 
2 x 5120 = 10240 shift operations to compute the required 1024-point convolution with an 
NTT based approach. Again, the convolution operation has to be repeated every R = 865 sam- 
ples. On a per-sample count we obtain 3.56 multiplications /sample, 13.03 additions /sample, 
and 11.84 shifts /sample. Considering also the one multiplication/ sample and the two addi- 
tions/sample for the separate computation of ||s„|| we obtain a total tally of 4.56 multiplica- 
tions/sample, 15.03 additions /sample, and 11.84 shifts/sample for the proposed approach. 

4. Conclusions 

We presented a. fast algorithm for the correlation computations that are required for the inven- 
tory based speech enhancement method proposed by Xiao et al. (Apr. 2009). The correlation 
computations are used in the inventory unit selection scheme of the enhancement procedure. 
They present a significant computational bottleneck for this method. The computational com- 
plexity of the inventory unit selection scheme would dominate the overall processing require- 
ment of the method by an order of magnitude if no fast algorithms were employed. 
The fast computation procedure proposed in this chapter is able to dramatically reduce the 
computational complexity of the proposed method without significantly affecting its enhance- 
ment performance. The number of multiplications per inventory sample required for the pro- 
cessing can be reduced from around 36.52 for a conventional FFT based method down to 
around 4.56 for the proposed NTT based method. The proposed approach is thus significantly 
faster than conventional computation methods. 
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1. Introduction 

DNA microarrays have become a tool of paramount importance in the study of gene func- 
tion, regulation, and interaction across large numbers of genes, and even entire genomes 
(Hegde et al., 2000; Moore, 2001). Microarray experiments generate pairs of 16 bits per pixel 
grayscale images (see Fig. 1, for an example). These images, which may require several tens 
of megabytes in order to be stored or transmitted, are analyzed by software tools that extract 
relevant information, such as the intensity of the spots and the background level. This infor- 
mation is then used for evaluating the expression level of individual genes (Hegde et al., 2000; 
Moore, 2001). 





(a) Green channel (b) Red channel 

Fig. 1. Example of a pair of images (1041 x 1044 pixels) that results from a microarray experi- 
ment. 

The common approach for microarray compression has been based on image analysis for spot 
finding (griding followed by segmentation) with the aim of separating the microarray image 
data into different streams based on pixel similarities (Adjeroh et al., 2006; Faramarzpour and 
Shirani, 2004; Faramarzpour et al, 2003; Hua et al., 2003; 2002; Jornsten et al., 2003; 2002a; 
Lonardi and Luo, 2004; Zhang et al., 2005). Once separated, the streams are compressed to- 
gether with the segmentation information. A potential drawback of these segmentation based 
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approaches is that different spot placements (e.g., non-rectangular) might compromise their 
performance. In fact, although initially the rectangular packing was the organization used for 
spot placement in microarrays, other non-rectangular packings have also been proposed (see 
Fig. 2). 





(a) Rectangular packing (b) Orange packing 

Fig. 2. Different spot packing: (a) Rectangular packing; (b) Orange packing (sample image 
from http://microarraylk.aecom.yu.edu/). Note that these images are not related. 
They serve just for illustrating different spot placements. 



Although initially most of the specialized techniques for microarray image compression con- 
sidered the lossy approach as a reasonable possibility (Faramarzpour and Shirani, 2004; Hua 
et al., 2003; 2002; Jornsten et al., 2003; 2002a), the most recent methods address mainly re- 
versible techniques (Faramarzpour et al., 2003; Lonardi and Luo, 2004; Zhang et al., 2005). 
Keeping the original images allows future re-analysis by possibly better algorithms. In fact, 
the analytic methods that are used for extracting information from the images are continu- 
ously being improved (Kothapalli et al., 2002; Leung and Cavalieri, 2003; Sasik et al., 2004). 
Also, as with other biomedical related data, legal issues might play a key role when choosing 
between maintaining or deleting the original data. 

Recently, we have investigated methods for compressing microarray images that do not re- 
quire spot segmentation. This new approach is based on arithmetic coding that is driven by 
image-dependent multi-bitplane finite-context models. Basically, the image is compressed on 
a bitplane basis, going from the most significant to the least significant bitplane. The finite- 
context model used by the arithmetic encoder uses (causal) pixels from the bitplane under 
compression and also pixels from the bitplanes already encoded. To our knowledge, this 
technique is currently the best one available in terms of compression efficiency of microarray 
images (Neves and Pinho, 2009). 

In this chapter, we start by describing the most important techniques for the lossless com- 
pression of microarray images that have been proposed in the literature. Then, we present a 
set of experiments that have been performed with the aim of providing a reference regard- 
ing the performance of standard image coding techniques, namely, lossless JPEG2000, JBIG 
and JPEG-LS, when applied to the lossless compression of microarray images. We proceed 
with the description of an image-independent multi-bitplane finite-context approach and we 
continue with the image-dependent version. Finally, we present experimental results that 
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illustrate the compression performance of the several approaches and we draw some conclu- 
sions. 

2. Compression techniques for microarray images 

In this section, we present the most important methods for compression of microarray im- 
ages, namely, the works of Jornsten et al. (2003), Hua et al. (2002), Faramarzpour et al. (2003), 
Lonardi and Luo (2004) and Zhang et al. (2005). Although all the methods presented in this 
section address the microarray compression problem using different approaches, some of the 
processing steps are common and similar to the ones depicted in Fig. 3. 

All the methods start by segmenting the microarray images into regions of interest (ROIs) con- 
taining the spot and some surrounding background. Some methods go even further, separat- 
ing the spot area from the background. However, the segmentation algorithm used in each 
method is different. 



Microarray image 




Fig. 3. The common processing steps of the compression methods presented in this section. 



Through segmentation, it is possible to encode the spots and background separately. This 
is explicitly done in the works of Hua et al. (2003; 2002); Jornsten et al. (2003); Jornsten and 
Yu (2000; 2002); Jornsten et al. (2002a;b); Lonardi and Luo (2004), and more implicilty in the 
work of Faramarzpour and Shirani (2004); Faramarzpour et al. (2003), because, in this case, the 
separation between the spot area and the background is performed only when the sequence 
is entropy encoded. 

Almost all available methods have also a lossy compression version. These methods remove 
what is considered to be noise or redundant. Although this step sounds obvious, the question 
is "What should be considered noise or redundant?" Note that, in the context of microarray 
images, the background is very important for noise estimation, because the bias due to noise 
can be estimated and removed in the calculation of the gene expression level of each spot. 
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The technique proposed by Jornsten et al. (2003) is characterized by a first stage devoted 
to griding and segmentation. Using the approximate center of each spot, a seeded region 
growing is performed for segmenting the spots. The segmentation map is encoded using 
chain-coding, whereas the interior of the regions are encoded using a modified version of the 
LOCO-I algorithm (LOw Complexity LOssless Compression for Images, the algorithm be- 
hind the JPEG-LS coding standard), named SLOCO. Besides lossy-to-lossless capability, Jorn- 
sten' s technique allows partial decoding by means of independently encoded image blocks. 
Hua et al. (2002) presented a transform-based coding technique. Initially, a segmentation is 
performed using the Mann- Whitney algorithm and the segmentation information is encoded 
separately. Due to the thresholding properties of the Mann- Whitney algorithm, the griding 
stage is avoided. Then, a modified EBCOT (Embedded Block Coding with Optimized Trun- 
cation) (Taubman and Marcellin, 2002) for handling arbitrarily shaped regions is used for en- 
coding the spots and background separately, allowing lossy-to-lossless coding of background 
only (with the spots encoded in lossless mode) or both background and spots. 
The compression method proposed by Faramarzpour et al. (2003) starts by locating and ex- 
tracting the microarray spots, isolating each spot into an individual ROI. A spiral path is ad- 
justed to each of these ROIs, such that its center coincides with the center of mass of the spot. 
The idea is to transform the ROI into an one-dimensional signal with minimum entropy. Then, 
predictive coding is applied along this path, with a separation between residuals belonging to 
the spot area and those belonging to the background area. 

Lonardi and Luo (2004) proposed lossless and lossy compression algorithms for microarray 
images (MicroZip). The method uses a fully automatic griding procedure, similar to that 
of Faramarzpour 's method, for separating spots from the background (which can be lossy 
compressed). Through segmentation, the image is split into two streams: foreground and 
background. Then, for entropy coding, each stream is divided into two 8 bit sub-streams 
and arithmetic encoded, with the option of being previously processed by a Burrows- Wheeler 
transform. 

The method proposed by Adjeroh et al. (2006); Zhang et al. (2005) is based on PPAM (Pre- 
diction by Partial Approximate Matching). PPAM is an image compression algorithm which 
extends the PPM text compression algorithm, considering the special characteristics of natural 
images (Zhang et al., 2005). Initially, the microarray image is separated into background and 
foreground. Then, for each of these two components, the pixel representation is separated 
into its most significant and least significant parts. To compress the data, the most significant 
part is first processed by an error prediction scheme. The residuals are then encoded by the 
PPAM context model and encoder. The least significant part is encoded directly by the PPAM 
encoder and the segmentation information is saved without compression. 

3. Standard image compression methods 

JBIG, JPEG-LS and JPEG2000 are state-of-the-art standards for coding digital images. They 
have been developed with different goals in mind, being JBIG more focused on bi-level 
imagery, JPEG-LS dedicated to the lossless compression of continuous-tone images and 
JPEG2000 designed with the aim of providing a wide range of functionalities. 
The JBIG standard (Joint Bi-level Image Experts Group) was issued in 1993 by ISO/IEC (In- 
ternational Organization for Standardization / International Electrotechnical Commission) 
and ITU-T (Telecommunication Standardization Sector of the International Telecommunica- 
tion Union) for the progressive lossless compression of binary and low-precision gray-level 
images (typically, having less than 6 bits per pixel). The major advantages of JBIG over other 
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existing standards, such as FAX Group 3/4, are its capability of progressive encoding and its 
superior compression efficiency (Hampel et al., 1992; ISO/IEC, 1993; Netravali and Haskell, 
1995; Salomon, 2000). The core of JBIG is an adaptive context-based arithmetic encoder, rely- 
ing on 1024 contexts when operating in sequential mode or on low resolution layers of the pro- 
gressive mode, or 4096 contexts when encoding high resolution layers. More recently, a new 
version, named JBIG2, has been published (ISO/IEC, 2000b), introducing additional function- 
alities to the standard, such as multipage document compression, two modes of progressive 
compression, lossy compression and differentiated compression methods for different regions 
of the image (e.g., text or halftones) (Salomon, 2000). 

JPEG-LS was developed by the Joint Photographic Experts Group (JPEG) with the aim of pro- 
viding a low complexity lossless image standard that could be able to offer better compression 
efficiency than lossless JPEG (ISO/IEC, 1999; Taubman and Marcellin, 2002; Weinberger et al., 
2000). Part 1 of this standard was finalized in 1999. The core of JPEG-LS is based on the 
LOCO-I algorithm, that relies on prediction, residual modeling and context-based coding of 
the residuals. Most of the low complexity of this technique comes from the assumption that 
prediction residuals follow a two-sided geometric probability distribution and from the use of 
Golomb codes which are known to be optimal for this kind of distributions. Besides lossless 
compression, JPEG-LS also provides a lossy mode where the maximum absolute error can 
be controlled by the encoder. This is known as near-lossless compression or Loo -constrained 
compression. 

From the three image coding standards addressed in this section, JPEG2000 is the most recent 
one (ISO/IEC, 2000a; Taubman and Marcellin, 2002). Part 1 was published as an Interna- 
tional Standard in the year 2000. It is based on wavelet technology and EBCOT coding of 
the wavelet coefficients, providing very good compression performance for a wide range of 
bitrates, including lossless coding. Moreover, JPEG2000 allows the generation of embedded 
code streams, meaning that from a higher bitrate stream it is possible to extract lower bitrate 
instances without the need for re-encoding. This property is of fundamental importance for 
progressive transmission, for example, over slow communication channels. 
These three standard image encoders cover a great variety of coding approaches. In fact, 
whereas JPEG2000 is transform based, JPEG-LS relies on predictive coding, and JBIG relies 
on context-based arithmetic coding. This diversity in coding engines might be helpful for 
drawing conclusions regarding the appropriateness of each of these technologies for the case 
of microarray image compression. 

3.1 Compression performance of the standards 

Before trying to develop new compression methods, it is always useful to find out how ex- 
isting compression standards behave on the class of images of interest. Therefore, for per- 
forming that assessment, we collected microarray images from three different publicly avail- 
able sources: (1) 32 images that we refer to as the Apo Al set and which have been col- 
lected from http : //www . stat .berkeley. edu /users /terry/ z array /Html/ index . 
html (this set was previously used by Jornsten et al. (2003); Jornsten and Yu (2002)); (2) 14 im- 
ages forming the ISREC set which have been collected from http : //www. isrec.isb-sib. 
ch/DEA/module 8 /P5_chip_image/ images/; (3) three images previously used to testMi- 
croZip (Lonardi and Luo, 2004), which were collected from http://www.cs.ucr.edu/ 
-yuluo /Micro Zip/. 
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JBIG compression was obtained using version 1.6 of the JBIG Kit package 1 , with sequential 
coding (-q flag). JPEG2000 lossless compression was obtained using version 5.1 of the JJ2000 
codec with default parameters (lossless compression) 2 . JPEG-LS coding was obtained using 
version 2.2 of the SPMG JPEG-LS codec with default parameters 3 . For additional reference, 
we also give compression results using the popular compression tool GZIP (version 1.2.4). 
Table 1 shows the compression results, in number of bits per pixel (bpp), where the first group 
of images corresponds to the Apo AI set, the second to the ISREC set and the third one to 
the MicroZip image set. Image size ranges from 1000 x 1000 to 5496 x 1956 pixels, i.e., from 
uncompressed sizes of about 2 megabytes to more than 20 megabytes (all images have 16 bits 
per pixel). The average results presented take into account the different sizes of the images, 
i.e., they correspond to the total number of bits divided by the total number of image pixels. 



Image set 


Gzip 


JPEG2000 


JBIG 


JPEG-LS 


APO_AI 

ISREC 

Microzip 


12.711 
12.464 
11.434 


11.063 

11.366 

9.515 


10.851 

10.925 

9.297 


10.608 

11.145 

8.974 


Average 


12.273 


10.653 


10.393 


10.218 



Table 1. Compression results, in bits per pixel (bpp), using lossless JPEG2000, JBIG and JPEG- 
LS. For reference, results are also given for the popular compression tool GZIP. 

The total average results show that gains of about 13.2%, 15.3% and 16.7%, in relation to GZIP 
compression, are attained respectively for lossless JPEG2000, JBIG and JPEG-LS, showing the 
superiority of image coding techniques over general purpose data compression methods in 
the task of compressing images. The average results by image set show that JPEG-LS provides 
the highest compression in the case of the Apo AI and MicroZip images, whereas JBIG gives 
the best results for the ISREC set. Lossless JPEG2000 is always slightly behind these two. It 
is interesting to note that the set for which JBIG gave the best results is also the one requiring 
more bits per pixel for encoding. 

3.1 .1 Sensitivity to noise 

It has been noted by Jornsten et al. (2003) that, in general, the eight least significant bitplanes 
of cDNA microarray images are close to random and, therefore, incompressible. Since this 
fact may result in some degradation in the compression performance of the encoders, we 
decided to address this problem and to study the effect of noisy bitplanes in the compression 
performance of the standards. 

To perform this evaluation, we separated the images into a number p of most significant bit- 
planes and 16 — p least significant bitplanes. Whereas the p most significant bitplanes have 
been sent to the encoder, the 16 — p least significant bitplanes have been left uncompressed. 
This means that the bitrate of a given image is the sum of the bitrate generated by encoding 
the p most significant bitplanes plus the 16 — p bits concerning the bitplanes that have been 
left uncompressed. 



http : //www. cl . cam. ac . uk/~mgk2 5/ jbigkit /. 
2 http://jj2000.epfl.ch. 
3 The original website of this codec, http : / /spmg . ece . ubc . ca, is currently unavailable. However, it 

can be obtained from ftp : //www. ieeta . pt /~ap/ codecs / jpeg_ls_v2 . 2 . tar . gz. 
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Fig. 4. Influence of noisy bitplanes in the performance of the standard encoding methods. The 
the curves indicate the bitrate obtained when only a given number p of the most significant 
bitplanes are sent to the encoder, whereas the other 16 — p bitplanes are left uncompressed. 



Image set 


JPEG2000 


JBIG 


JPEG-LS 




8bp 


Best 


8bp 


Best 


8bp 


Best 


Apo_AI 
ISREC 

MicroZip 


10.940 
11.100 
9.918 


10.790 
10.954 
9.321 


10.510 
10.607 
9.506 


10.507 
10.583 
9.030 


10.523 
10.838 
9.588 


10.433 
10.713 
8.912 


Average 


10.661 


10.376 


10.224 


10.073 


10.302 


10.026 



Table 2. Average compression results, in bits per pixel (bpp), when a number of bitplanes 
is left uncompressed. The columns labeled "8 bp" provide results for the case where only 
the 8 most significant bitplanes have been encoded and the 8 least significant bitplanes have 
been left uncompressed. The column named "Best" contains the results for the case where the 
separation of most and least significant bitplanes has been optimally found. 
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Figure 4 depicts bitrate curves, as a function of p, for two different images, "1230clG" and 
"array 1". As can be observed, the best bitrate is generally not met when compressing all 
16 bitplanes, but instead when some of the least significant bitplanes are left uncompressed. 
However, the value of the optimum value of p, p pt/ varies not only from image to image, 
but also from one encoder to the other. In fact, for the Apo AI set, which is characterized 
by the most regular value of p op t, JBIG is the encoder with the highest value of p Q pt (around 
8), then comes lossless JPEG2000 (around 10) and, finally, JPEG-LS (around 13). This result 
is not surprising, since JBIG encodes the bitplanes independently. Therefore, without being 
able to get information from other bitplanes, it is natural that JBIG starts considering bitplanes 
as "noise" earlier than the other encoders. Moreover, this can also be the justification for its 
better performance in the ISREC set, because it is the most noisy. 

Table 2 compares average results for the three set of images regarding two situations: (1) 
the image is divided into the eight most significant bitplanes (which are encoded) and the 
eight least significant bitplanes (which are left uncompressed); (2) the optimum value of p 
is determined for each image. From this table, and comparing with the Table 1, we can see 
that, in fact, this splitting operation can provide some additional compression gains. The best 
results attained provided improvements of 3.1%, 2.6% and 1.9% respectively for JBIG, lossless 
JPEG2000 and JPEG-LS. 

However, finding the right value for p may require as many as 16 iterations of the compression 
phase in order to find it. Moreover, from the results shown in Table 2, we can see that a simple 
separation of the bitplanes in an upper and lower half may improve the compression in some 
cases (Apo AI and ISREC image sets), but may also produce the opposite result (MicroZip 
image set). 

3.1.2 Lossy-to-lossless compression 

From the point of view of compression efficiency, and taking into account the results pre- 
sented in Table 1, JPEG-LS is the overall best lossless compression method, followed by JBIG 
and lossless JPEG2000. The difference between JPEG-LS and lossless JPEG2000 is about 4.1% 
and between JPEG-LS and JBIG is only 1.7%. However, the better compression performance 
provided by JPEG-LS can be overshadowed by a potentially important functionality provided 
by the other two standards, which is progressive, lossy-to-lossless, transmission. 
In the case of lossless JPEG2000, this functionality is basically a by-product of the multi- 
resolution wavelet technology used in its encoding engine and also due to a strategy of en- 
coding the information in layers (Taubman and Marcellin, 2002). In the case of JBIG, this 
property comes from two different sources. On one hand, images with more that one bitplane 
are encoded using a bitplane-by-bitplane coding approach. This provides a kind of progres- 
sive transmission, from most to least significant bitplanes, where the precision of the pixels 
is improved for each added bitplane. Moreover, this technique produces a reduction of the 
Loo error by a factor of two for each additional bitplane. On the other hand, JBIG permits the 
progressive transmission of each bitplane by progressively increasing its spatial resolution 
(ISO/IEC, 1993; Salomon, 2000). However, the compression results that we present in Table 1 
do not take into account the additional overhead implied by this encoding mode of JBIG (we 
used the -q flag of the encoder, which disables this mode). 

In Fig. 5, we present rate-distortion curves for two images, "1230clG" and "arrayl", obtained 
with the lossless JPEG2000 and JBIG coding standards, and according to two error metrics: 
L2~norm (root mean squared error) and Loo -norm (maximum absolute error). Regarding the 
L2-norm, we observe that lossless JPEG2000 provides slightly better rate-distortion results for 
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JBIG: 1230C1G 

JPEG2000: 1230clG 

JBIG: arrayl 

JPEG2000: arrayl 




Bitrate (bpp) 

Fig. 5. Rate distortion curves showing the performance of lossless JPEG2000 and JBIG in a 
lossy-to-lossless mode of operation. Results are given both for the L^ (root mean squared 
error) and Loo (maximum absolute error) norms. 



bitrates less than 8 bpp. For higher bitrates, this codec exhibits a sudden degradation of the 
rate-distortion. We believe that this phenomenon is related to the default parameters used by 
the encoder, which might not be well suited for images having 16 bits per pixel, such as those 
of the microarrays. Moreover, we think that a careful setting of these parameters may lead to 
improvements in the rate-distortion of JPEG2000 for bitrates higher than 8 bpp, although we 
consider this tuning a problem that is beyond the scope of this work. 

With respect to the Loo -norm, we observe that JBIG is the one with the best rate-distortion 
performance. In fact, due to its bitplane-by-bitplane approach, it guarantees an exponential 
and upper bounded decrease of the maximum absolute error. The upper bound of the error is 
given by 2( 16 ~P) — 1, where p is the number of bitplanes already decoded. Contrarily, lossless 
JPEG2000 cannot guarantee such bound, which may be a major drawback in some cases. Fi- 
nally, we note that the sudden deviation of the lossless JPEG2000 curves around bitrates of 8 
bpp is probably related to the same problem pointed out earlier for the case of the L2-norm. 



438 Signal Processing 



3.2 Conclusions 

The main objective of this section was to provide a set of comprehensive results regarding 
the lossless compression of microarray images by state-of-the-art image coding standards, 
namely, lossless JPEG2000, JBIG and JPEG-LS. In order to facilitate future comparisons by 
other researchers, we collected a total of 49 microarray images available from the Internet. We 
believe that the development of specialized compression techniques should be supported by a 
preliminary study of the performance provided by well established methods and, particularly, 
by those that are standards. Only after making such study it is possible to be in a comfortable 
position for arguing about the relevance of some specialized technique. 

From the experimental results obtained, we conclude that JPEG-LS gives the best lossless com- 
pression performance. However, it lacks lossy- to-lossless capability, which may be a decisive 
functionality if remote transmission over possibly slow links is a requirement. Complying to 
this requirement we find JBIG and lossless JPEG2000, lossless JPEG2000 being the best consid- 
ering rate-distortion in the sense of the L2-norm and JBIG the most efficient when considering 
the Loo -norm. Moreover, JBIG is consistently better than lossless JPEG2000 regarding lossless 
compression ratios. Also, JBIG is the method that can benefit most from a correct separa- 
tion of most significant bitplanes that are encoded and least significant bitplanes that are left 
uncompressed (it gained 3.1%), and it is also the coding technique that, due to the bitplane- 
by-bitplane coding, can search for the optimum point of separation on-the-fly In fact, this can 
be done by monitoring the bitrate resulting from the compression of each bitplane, and stop 
doing compression when this value is over 1 bpp. As a final conclusion, and according to 
what we presented in this section, it is our opinion that the technology behind JBIG seems to 
be the most appropriate for microarray image coding. 

4. Compression of microarray images using finite-context models and arithmetic 
coding 

4.1 Finite-context models 

The core of the methods proposed in the remainder of this chapter consists of an adaptive 
finite-context model followed by arithmetic coding. A finite-context model (see Fig. 6) of an 
information source assigns probability estimates to the symbols of an alphabet A, according to 
a conditioning context computed over a finite and fixed number, M, of past outcomes (order- 
M finite-context model) (Rissanen, 1983; Rissanen and Langdon, Jr., 1981; Sayood, 2000). At 
time t, we represent these conditioning outcomes by c l — Xt-M+l/ • • • / x>t-\, %t> The number of 
conditioning states of the model is |*4| M , dictating its complexity (or model cost). In our case, 
A = {0, 1} and, therefore, \A\ = 2. 
In practice, the probability that the next outcome, x t +i, is "0" is obtained using the estimator 

^'-*>- . ( j)+%, + / )+a - m 

where n^s,^) represents the number of times that, in the past, the information source gen- 
erated symbol s e A having c l as the conditioning context. The parameter 8 > 0, besides 
allowing fine tuning the estimator, avoids generating zero probabilities when a symbol is en- 
coded for the first time. In our case, we used 5 = 1, which corresponds to Laplace's estimator 
(it can be seen as an initialization of all counters to one). The counters are updated each time 
a symbol is encoded. Since the context template is causal, the decoder is able to reproduce the 
same probability estimates without needing additional information. 
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Fig. 6. Finite-context model: the probability of the next outcome, x t +\, is conditioned by the 
M last outcomes. In this example, M = 5. 



Context, c l 


n(oy) 


n(iy) 


ft(oy) + n(iy) 


00000 


23 


41 


64 


00001 


16 


6 


22 


00010 


19 


30 


49 


00011 


34 


42 


76 


00100 


36 


17 


53 


11111 


8 


2 


10 



Table 3. Simple example illustrating how finite-context models are implemented. The rows 
of the table represent a probability model at a given instant t. In this example, the particular 
model that is chosen for encoding a symbol depends on the last five encoded symbols (order-5 
context). 



Table 3 shows an example of how a finite-context is typically implemented. In this example, 
an order-5 finite-context model is presented. Each row represents a probability model that 
is used to encode a given symbol according to the last encoded symbols (five in this exam- 
ple). Therefore, if the last symbols were "00010", i.e., c l = 00010, then the model communi- 
cates the following probability estimates to the arithmetic encoder: P(0|00010) = 19/49 and 
P(l|00010) = 30/49. 

The block denoted "Encoder" in Fig. 6 is an arithmetic encoder. It is well known that prac- 
tical arithmetic coding generates output bit-streams with average bitrates almost identical to 
the entropy of the model (Bell et al., 1990; Salomon, 2000; Sayood, 2000). In our case, the 
theoretical bitrate average (entropy) of the model after encoding N symbols is given by 



N-l 



Hn 



N 



E lo g2 p (*m = s \ ct ) b P s ' 



(2) 



f=0 



where "bps" stands for "bits per symbol". Since we are dealing with images, instead of using 
the generic "bps" measure we use "bpp", which stands for "bits per pixel". Recall that the 
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entropy of any sequence of two symbols is limited to 1 bps, a value that is achieved when the 
symbols are independent and equally likely 

4.2 Image-independent contexts 

In Section 3, we presented a study of the compression performance of three image coding stan- 
dards in the context of microarray image compression: JPEG2000, JBIG and JPEG-LS. Since 
they rely on three different coding technologies, we were able not only to evaluate the perfor- 
mance of each of these standards, but also to collect hints regarding what might be the best 
coding technology regarding microarray image compression. In that study, we concluded 
that from the three technologies evaluated (predictive coding in the case of JPEG-LS, trans- 
form coding in the case of JPEG2000 and context-based arithmetic coding in the case of JBIG), 
the technology behind JBIG seemed to be the most promising. In fact, JPEG-LS provided the 
highest compression, closely followed by JBIG. However, unlike JPEG2000 and JBIG, it does 
not provide lossy-to-lossless capabilities, a characteristic that might be of high interest, spe- 
cially in the case where remote databases have to be accessed using transmission channels of 
reduced bandwidth. Moreover, with JBIG, the image bitplanes are compressed independently, 
suggesting the existence of some room for improvement. 

Motivated by these observations, we developed a compression method for microarray images 
which is based on the same technology as JBIG but that, unlike JBIG, exploits inter-bitplane 
dependencies, providing coding gains in relation to JBIG (Neves and Pinho, 2006). Designing 
contexts that gather information from more than one bitplane (multi-bitplane contexts) is not 
just a matter of joining more bits to the context, because for each new bit added the memory 
required doubles. Moreover, there is the danger of running into the context dilution problem, 
due to the lack of sufficient data for estimating the probabilities. Therefore, this extension to 
multi-bitplane contexts must be done carefully. 

The method proposed by Neves and Pinho (2006) was inspired by EIDAC (Yoo et al., 1998), a 
compression method that has been used with success for coding images with a reduced num- 
ber of intensities (simple images). The images are compressed on a bitplane basis, from the 
most to the least significant bitplane. The causal finite-context model that drives the arithmetic 
encoder uses pixels both from the bitplane currently being encoded and from the bitplanes 
already encoded. As encoding proceeds, the average bitrate obtained after encoding each bit- 
plane is monitored. If, for some bitplane, the average bitrate exceeds one bit per pixel, then 
the encoding process is stopped and the remaining bitplanes are saved without compression. 
The encoding procedure is outlined in Fig. 7. 

The context modeling part of EIDAC was designed mainly with the aim of compressing im- 
ages with eight bitplanes or less, implying, at most, 19 bits of context. A straightforward ex- 
tension to images with 16 bitplanes would require contexts of 27 bits, i.e., at least 2 x 2 27 = 2 28 
counters. Essentially, the technique proposed by Neves and Pinho (2006) differs from EIDAC 
in three aspects: (1) it was designed taking into account the specific nature of the images, 
keeping the size of the contexts limited to 21 bits; (2) it does not use the histogram packing 
procedure proposed for EIDAC because, generally, microarray images have dense intensity 
histograms; (3) it implements a rate-control mechanism that avoids producing average bitrates 
of more than one bit per pixel in bitplanes that are too noisy (this is a common characteristic 
of the least significant bitplanes of microarray images (Jornsten et al., 2003)). 
As we mentioned before, choosing the context template for a multi-bitplane image is a critical 
task, requiring tradeoffs involving aspects such as the maximum size of the context, the prob- 
lem of context dilution and the placement of the context bits such that the maximum informa- 
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Encode bitplane k 
k=15, 14, 13, ..., 



Choose the best 
context size 



Encode all pixels 
in the bitplane 




No 



Save the remaining 
bitplanes uncompressed 



Fig. 7. Encoding procedure of the method proposed by Neves and Pinho (2006). The choice of 
the context shape is based on Fig. 8. Note that, being a bitplane based encoder, it is possible 
to monitor the bitrate used to encode each bitplane. 



tion can be collected. This work was done in (Neves and Pinho, 2006) mainly using a trial and 
error procedure, leading to the image-independent context configuration displayed in Fig. 8. 
Note that, when encoding the eight least significant bitplanes, the finite-context model is only 
formed with pixels from the higher numbered bitplanes. This specific context configuration 
together with the rate-control mechanism avoids the degradation in compression rate when 
there are bitplanes that are close to random and, therefore, are almost incompressible. 
Although being able to provide state-of-the-art compression results, the method proposed in 
(Neves and Pinho, 2006) could be improved. In fact, due to its image-independent nature, 
and despite being designed for a specific type of images (microarrays), the context configura- 
tion depicted in Fig. 8 resulted from a complicated process that tried to balance the inevitable 
particularities among the images. From the point of view of a single image, this context con- 
figuration might seem overkill, i.e., a smaller context might suffice. However, it is needed for 
satisfying the ensemble of images. This observation motivated the image-dependent context- 
modeling approach that we describe in the next section. 
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(a)BP = 15 




(d)8<=BP<=12 



(e) <= BP <= 7 

Fig. 8. Image-independent context configuration used in (Neves and Pinho, 2006) at five dif- 
ferent compression stages: (a) when encoding the most significant bitplane (four bits of con- 
text); (b) when encoding the second most significant bitplane (ten bits of context); (c) when 
encoding the third most significant bitplane (16 bits of context); (d) from the fourth until the 
eighth most significant bitplanes (17-21 bits of context); (e) the eight least significant bitplanes 
(13-20 bits of context). Context positions falling outside the image at the image borders are 
considered as having zero value. 



5. Image-dependent finite-context models 

Instead of using the image-independent context model presented in Fig. 8, the algorithm that 
we describe in this section tries to find the "best" context configuration to encode the current 
bitplane, based on the templates depicted in Fig. 9. The test of all possible context configura- 
tions is a hard task, virtually impossible, due to the huge number of possibilities. To overcome 
this drawback, we developed a greedy approach that we explain next. 
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(a) (b) 

Fig. 9. (a) The template used for growing the context at the level of the bitplane currently 
being encoded; (b) The template used for growing the context corresponding to the bitplanes 
already encoded. 



Before encoding a bitplane, p, the algorithm constructs an appropriate context configuration 
through an iterative process (note that bitplanes are numbered from 0, the least significant 
bitplane, to 15, the most significant bitplane). In each iteration, an additional context bit is 
tested in each of the 16 — p possible locations, one in each of the 16 — p context bitplanes that 
are available. In a given context bitplane, the additional bit can only be inserted in position k 
of the corresponding template displayed in Fig. 9 if all positions i < k belong already to the 
best context configuration found so far. This means that the part of the context belonging to a 
given bitplane can grow only according to the pixel numbering shown in Fig. 9. The template 
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in Fig. 9(a) is used for the context bitplane p, whereas the template in Fig. 9(b) applies to the 
remaining context bitplanes, i.e., from bitplane p + 1 to bitplane 15. 

After performing an iteration, the new context bit is assigned to the position where the largest 
improvement in the compression performance of bitplane p occurred. If none of the possi- 
ble 16 — p context bit positions were able to improve the compression, then the search stops 
and the context configuration found so far is used for encoding the bitplane p. Otherwise, a 
new context bit is tested. This iterative process proceeds while the new context bit is able to 
improve the compression performance of bitplane p or until the maximum context depth is 
reached. For the results presented in this section, we used a maximum of 20 context bits. Fig- 
ure 10 presents an example of the context configuration obtained with this process for some 
of the bitplanes of the image "1230clG" (APO_AI image set). 

The configuration of the context bits for a particular bitplane, p, can be communicated to the 
decoder using approximately 4(16 — p) bits. Note that the maximum number of context bits 
per context bitplane is less that 16 (see Fig. 9) and, therefore, can be represented in four bits. 
Hence, the total overhead regarding the image-dependent contexts is just some tens of bytes. 
The algorithm is outlined next. 

bestCtx := 0-order context; 

bestRate := rate for encoding bitplane 

with 0-order context; 
do 

improved := FALSE; 

for p := bitplane to be encoded, 15 
if p = bitplane to be encoded 

add bit according to Fig. 9(a); 
else 

add bit according to Fig. 9(b); 
end 
rate := rate for encoding bitplane 

using current context; 
if rate < bestRate 

bestCtx := current context; 
bestRate := rate; 
improved := TRUE; 
end 

remove bit added above; 
end 
while size of bestCtx < 20 AND improved 

Being a greedy approach, it is not guaranteed that the optimum is found. In fact, as can be 
seen, for each context bitplane the context can only grow according to a predefined order 
which is given by the pixel numbering associated to the templates of Fig. 9. This limits the 
number of degrees of freedom of the search process, reducing the probability of finding the 
optimum configuration, but, on the other hand, also allowing running this procedure in a 
reasonable time. 

In order to further accelerate the process of choosing these image-dependent contexts, and 
due to the highly structured nature of microarray images, we developed another version of 
the algorithm where only a small region of the image is used for constructing the contexts. 
Using this faster approach, the results obtained for a region of 256 x 256 pixels have been 
slightly worse. However, we verified a significant reduction in the time spent. In a 2 GHz 
Pentium 4 computer with 512 MBytes of memory, the MicroZip test set (three images totaling 
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(b)BP=13 



(e)BP = 10 

Fig. 10. Context configuration obtained by the proposed method in five different bitplanes of 
the image "1230clG": (a) when encoding bitplane 14 (seven bits of context); (b) when encoding 
bitplane 13 (11 bits of context); (c) when encoding bitplane 12 (13 bits of context); (d) when 
encoding bitplane 11 (17 bits of context); (e) when encoding bitplane 10 (20 bits of context). 
Context positions falling outside the image at the image borders are considered as having zero 
value. 



approximately 21 million pixels) required about 220 minutes to compress when the whole 
image was used to performed the search. When we used a region of 256 x 256 pixels, it 
required approximately 6 minutes to compress the MicroZip test set (about 2 minutes more 
than the image-independent approach). These three images have sizes of 1916 x 1872, 5496 x 
1956 and 3625 x 1929 pixels. Decoding is faster, because the decoder does not have to search 
for the best context: that information is embedded in the bitstream. 

6. Experimental results 

Table 4 shows the average compression results, in bits per pixel, for the three sets of images 
described previously (see Section 3). In this table, we present experimental results of both the 
image-independent and the image-dependent approaches. We also include results obtained 
with SPIHT (Said and Pearlman, 1996) 4 and EIDAC (Yoo et al, 1998). 

Comparing with the results presented in Table 1, we can see that the fast version of the image- 
dependent method (indicated as "256 x 256" in the table) is 6.3% better than JBIG, 4.7% bet- 
ter than JPEG-LS and 8.6% better than lossless JPEG2000. It is important to remember that 
JPEG-LS does not provide progressive decoding, a characteristic that is intrinsic to the image- 
dependent multi-bitplane finite-context method and also to JPEG2000 and JBIG. From the re- 
sults presented in Table 4, it can also be seen that using an area of 256 x 256 pixels in the center 
of the image for finding the context, instead of the whole image, leads to a small degradation 
in the performance (about 0.3%), showing the appropriateness of this approach. 



4 SPIHT codec from http: //www. cipr . rpi . edu/research/SPIHT/ (version 8.01). 
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Image set 


SPIHT 


EIDAC 


Image 
independent 


Image-dependent 




256x256 


Full 


APO_AI 

ISREC 

MicroZip 


10.812 

11.098 

9.198 


10.543 
10.446 

8.837 


10.280 

10.199 

8.840 


10.225 

10.198 

8.667 


10.194 

10.158 

8.619 


Average 


10.378 


10.005 


9.826 


9.741 


9.708 



Table 4. Average compression results, in bits per pixel, using SPIHT, EIDAC, the image- 
independent and the image-dependent methods. The "256 x 256" column indicates results 
obtained with a context model adjusted using only a square of 256 x 256 pixels at the center 
of the microarray image, whereas "Full" indicates that the search was performed in the whole 
image. The average results presented take into account the different sizes of the images, i.e., 
they correspond to the total number of bits divided by the total number of image pixels. 



Table 5 confirms the performance of the image-dependent method relatively to two recent 
specialized methods for compressing microarray images: MicroZip (Lonardi and Luo, 2004) 
and Zhang's method (Adjeroh et al., 2006; Zhang et al., 2005). As can be observed, the image- 
dependent multi-bitplane finite-context method provides compression gains of 9.1% relatively 
to MicroZip and 6.2% in relation to Zhang's method, on a set of test images that has been used 
by all these methods. 



Images 


MicroZip 


Zhang 


Image 
independent 


Image-dependent 




256x256 


Full 


arrayl 
array2 
array3 


11.490 

9.570 
8.470 


11.380 
9.260 
8.120 


11.105 
8.628 
7.962 


11.120 
8.470 
7.717 


11.056 
8.423 
7.669 


Average 


9.532 


9.243 


8.840 


8.667 


8.619 



Table 5. Compression results, in bits per pixel, using two specialized methods, MicroZip 
and Zhang's method, the image-independent method and the image-dependent method. The 
"256 x 256" column indicates results obtained with a context model adjusted using only a 
square of 256 x 256 pixels at the center of the microarray image, whereas "Full" indicates that 
the search was performed in the whole image. 



Figure 11 shows, for three different images, the average number of bits per pixel that are 
needed for representing each bitplane. As expected, this value generally increases when 
going from most significant bitp lanes to least significant bitp lanes. For the case of images 
"Def 6 61Cy3" and "12 30clG", it can be seen that the average number of bits per pixel re- 
quired by the eight least significant bitplanes is close to one, as pointed out by Jornsten et al. 
(2003). However, image "array 3" shows a different behavior. Because this image is less 
noisy, the compression algorithm is able to exploit redundancies even in lower bitplanes. This 
is done without compromising the compression efficiency of noisy images, due to the mech- 
anism that monitors and controls the average number of bits per pixel required for encoding 
each bitplane. 

The maximum number of context bits that we allowed for building the contexts was limited 
to 20. Since the coding alphabet is binary, this implies, at most, 2 x 2 20 = 2097152 counters 
that can be stored in approximately 8 MBytes of computer memory. In a 2 GHz Pentium 4 
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Def661Cy3 

1230clG 

array3 




Bitplane 

Fig. 11. Average number of bits per pixel required for encoding each bitplane of three different 
microarray images (one from each test set). 



computer with 512 MBytes of memory, the image-dependent algorithm required about six 
minutes to compress the MicroZip test set (note that this compression time is only indicative, 
because the code has not been optimized for speed). Decoding is faster, because the decoder 
does not have to search for the best context. Just for comparison, the codecs of the compression 
standards took approximately one minute to encode the same set of images. 

7. Conclusions 

The use of microarray expression data in state-of-the-art biology has been well established. 
The widespread adoption of this technology, coupled with the significant volume of data gen- 
erated per experiment, in the form of images, has led to significant challenges in storage and 
query-retrieval. In this work, we have studied the problem of coding this type of images. 
We presented a set of comprehensive results regarding the lossless compression of microar- 
ray images by state-of-the-art image coding standards, namely, lossless JPEG2000, JBIG and 
JPEG-LS. From the experimental results obtained, we conclude that JPEG-LS gives the best 
lossless compression performance. However, it lacks lossy-to-lossless capability, which may 
be a decisive functionality if remote transmission over possibly slow links is a requirement. 
Complying to this requirement we find JBIG and lossless JPEG2000, lossless JPEG2000 being 
the best considering rate-distortion in the sense of the L2-norm and JBIG the most efficient 
when considering the Loo -norm. Moreover, JBIG is consistently better than lossless JPEG2000 
regarding lossless compression ratios. 

Motivated by these findings, we have developed efficient methods for lossless compression 
of microarray images, allowing progressive, lossy-to-lossless decoding. These methods are 
based on bitplane compression using image-independent or image-dependent finite-context 
models and arithmetic coding. They do not require griding and /or segmentation as most 
of the specialized methods that have been proposed do. This may be an advantage if only 
compression is sought, since it reduces the complexity of the method. Moreover, since they 
do not require griding, they are robust, for example, against layout changes in spot placement. 
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The results obtained by the multi-bitplane context-based methods have been compared with 
the three image coding standards and with two recent specialized methods: MicroZip and 
Zhang's method. The results obtained show that these new methods have better compression 
performance in all image test sets used. 
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1. INTRODUCTION 

Due to the finite precision nature of computer arithmetic, the output roundoff noise of a fixed- 
point IIR digital filter usually arises. This noise is critically dependent on the internal structure 
of an IIR digital filter [1],[2]. Error feedback (EF) is known as an effective technique for reduc- 
ing the output roundoff noise in an IIR digital filter [3]-[5]. Williamson [6] has reduced the 
output roundoff noise more effectively by choosing the filter structure and applying EF to the 
filter. Lu and Hinamoto [7] have developed a jointly optimized technique of EF and realiza- 
tion to minimize the effects of roundoff noise at the filter output subject to /2-norm dynamic- 
range scaling constraints. Li and Gevers [8] have analyzed the output roundoff noise of the 
closed-loop system with a state-estimate feedback controller, and presented an algorithm for 
realizing the state-estimate feedback controller with minimum output roundoff noise under 
/2-norm dynamic-range scaling constraints. Hinamoto and Yamamoto [9] have proposed a 
method for applying EF to a given closed-loop system with a state-estimate feedback con- 
troller. 

This paper investigates the problem of jointly optimizing EF and realization for the closed- 
loop system with a state-estimate feedback controller so as to minimize the output roundoff 
noise subject to /2-norm dynamic-range scaling constraints. To this end, the problem at hand is 
converted into an unconstrained optimization problem by using linear-algebraic techniques, 
and then an iterative technique which relies on a quasi-New ton algorithm [10] is developed. 
With a closed-form formula for gradient evaluation and an efficient quasi-Newton solver, the 
unconstrained optimization problem can be solved efficiently. Our computer simulation re- 
sults demonstrate the validity and effectiveness of the proposed technique. 
Throughout the paper, I n stands for the identity matrix of dimension n x n, the transpose 
(conjugate transpose) of a matrix A is indicated by A T (A*), and the trace and zth diagonal 
element of a square matrix A are denoted by tr[A] and (A) a, respectively. 

2. ROUNDOFF NOISE ANALYSIS 

Consider a stable, controllable and observable linear discrete-time system described by 

x(k + l) = A x(k) + b u(k) 

(1) 

y(k) = c x(k) 
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where x(k) is an n x 1 state-variable vector, u(k) is a scalar input, y(k) is a scalar output, and 
A 0r b and c are n x n, n x 1 and 1 x n real constant matrices, respectively. The transfer 
function of the linear system in (1) is given by 

H (z) = c (zI n -A )- 1 b . (2) 

If a regulator is designed by using the full-order state observer, we obtain a state-estimate 
feedback controller as 

x(k + 1) = F x(k) + M(fc) + £ y(fc) 



= RoSt(k) + b r(k)+g y(k) 

u(k) = -k x(k)+r(k) 



(3) 



where x(/c) is an n x 1 state-variable vector in the full-order state observer, g is an n x 1 gain 
vector chosen so that all the eigenvalues of F = A — g c are inside the unit circle in the 
complex plane, k is a 1 x n state-feedback gain vector chosen so that each of the eigenvalues 
of A — b k is at a desirable location within the unit circle, r(k) is a scalar reference signal, 
and R = F — b k . The closed-loop control system consisting of the linear system in (1) and 
the state-estimate feedback controller in (3) is illustrated in Fig. 1. 
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Fig. 1. The closed-loop control system with a state-estimate feedback controller. 



When performing quantization before matrix-vector multiplication, we can express the finite- 
word-length (FWL) implementation of (3) with error feedback as 



where 



x(k + 1) = R Q[x(k)] + br(k) + gy(k) + De(k) 
u(k) = -kQ[x(k)}+r(k) 

e(k) = x(k) - Q[x(k)} 



(4) 



is an n x 1 roundoff error vector and D is an n x n error feedback matrix. All coefficient 
matrices R, b, g and k are assumed to have an exact fractional B c bit representation. The FWL 
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state-variable vector x(k) and signal u(k) all have a B bit fractional representation, while the 
reference input r(k) is a (B — B c ) bit fraction. The vector quantizer Q[-] in (4) rounds the B 
bit fraction x(k) to (B — B c ) bits after completing the multiplications and additions, where the 
sign bit is not counted. It is assumed that the roundoff error vector e(k) can be modeled as a 
zero-mean noise process with covariance (T 2 I n where 



a 1 



12 



-2(1 



It is noted that if the z'th element of the roundoff error vector e(k) is indicated by ej(k) for i = 
1, 2, • • • , n then the variable e z - (k) can be approximated by a white noise sequence uniformly 
distributed with the following probability density function: 



P(ei(k)) 



for 



c) < e t (k) < l2-( B - B ^) 
otherwise 




Q[x(k)] 



Fig. 2. A state-estimate feedback controller with error feedback. 



The closed-loop system consisting of the linear system in (1) and the state-estimate feedback 
controller with error feedback in (4) is shown in Fig. 2, and is described by 



x(k + l) 
x{k + l) 



y{k) 



x(k) 
x(k)_ 



+ br(k)+Be(k) 



(5) 
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where 

b 

b 

[Co 0]. 

From (5), the transfer function from the roundoff error vector e(k) to the output y(k) is given 
by 

G D (z)=c(zI 2n -A)- 1 B. (6) 

The output noise gain }(D) = a^ ut /a 2 is then computed as 

/(D) = tr[W D ] (7) 

with 

Wd= ^jL g ° (2)Gd(2) I (8) 

where o^ ut stands for the noise variance at the output. For tractability, we evaluate J(D) in (7) 
by replacing R, b, g and k by R , b , g and k , respectively Defining 



In 

In 

the transfer function in (6) can be expressed as 

-l- 



G D (z) = cS(zI 2n ~ S- l AS)- l S- l B 





-In 



(9) 



: C(zl 2n ~ <*>) 



b k 
F -D 

■ c (zl n - A + b k )~ 1 b k (zl n 
izIn-D) 
■c(zl ln - ty-illizln - D) 



(10) 



where 



O 



U 



A. UqKo UqICo 

F n 


In 



It is noted that the stability of the closed-loop control system is determined by the eigenvalues 
of matrix A in (5), or equivalently, those of matrix <t> in (10). This means that neither of the 
roundoff error vector e(k) and the error-feedback matrix D affects the stability. 
Substituting (10) into matrix Wp in (8) gives 



W D = (bokofW^oko + (b k ) T W 2 (F - D) 
+ (F - D) T W 3 b k 
+ (F - D) T W 4 (F - D) 



(11) 
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W 



where 

W = <D T W(D + c T c 

' W 1 w 2 ' 

w 3 w 4 

Since W is positive semidefinite, it can be shown that there exists an n x n matrix P such that 
W3 = W4P. In addition, (11) can be written by virtue of W 2 = Wj as 

W D = (F + P^O - D) T W 4 (F + P^q/co - D) 
+ (b /co) T (W 1 -P T W 4 P)bofco- 



(12) 



Alternatively, applying z-transform to the first equation in (5) under the assumption that 
e(k) = 0, we obtain 

X(2) 



X(z) 



(zl- 



^^(z) 



(13) 



where X(z), X(z) and R(z) represent the z-transforms of x(/c), x(k) and r(/c), respectively 
Replacing R, b, k and ^ by R 0/ b 0/ k and g , respectively, and then using 



X(z) 
X(z) 



(z^-S-'AS)"^" 1 ?? 



yields 
where 



X(z) = X(z) = F(z)jR(z) 



(14) 



F(z) = [zI n -(A -b k )}- 1 b . 
The controllability Gramian K defined by 

K=-^-J F(z)F*(z) — 

2tt/ J\ z \=i w v ; z 

can be obtained by solving the following Lyapunov equation: 

K=(A - b k )K(A - b k ) T + b b T . 
3. ROUNDOFF NOISE MINIMIZATION 



(15) 



(16) 



Consider the system in (4) with D = and denote it by (R, b,g, k) n . By applying a coordinate 
transformation x f (k) = T~ 1 x(k) to the above system (R, b,g, k) n , we obtain a new realization 
characterized by (R,b,g,k) n where 



R = T 1 RT / b = T 1 b 
g=T~ 1 g / k = kT. 

For the system described by (17), the counterparts of W,- for i = 1,2,3,4 are given by 

W, = T T W,T 



(17) 



(18) 
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and the corresponding output noise gain is given by 

J(D,T)=tr[W D ] (19) 

where Wp can be obtained referring to (11) as 

w D = [r- 1 (F + Pb fc )r-D] T 

■T T W 4 T [j- 1 (f o + Pb k )T - D] 
+ T T (b k ) T (W 1 - P T W i P)b k T. 
In addition, (15) can be written as 

K=J-,I T- 1 F(z)F*(z)T- T — 

2nj J| 2 |=i 2 (20 ) 



As a result, the output roundoff noise minimization problem amounts to obtaining matrices 
D and T which jointly minimize /(D, T) in (19) subject to the /2-norm dynamic-range scaling 
constraints specified by 

(*),-,■ = (T^KT-^u = 1, i = l,2,---,n. (21) 

To deal with (21), we define 

t=T T K-2. (22) 

Then the /2-norm dynamic-range scaling constraints in (21) can be written as 

(t~ T t~ 1 ) ii = l, i = l,2 r --,n. (23) 

These constraints are always satisfied if f assumes the form 



t" 1 



t n 



tn 



(24) 



(25) 



Substituting (22) into (19), we obtain 

]{B f t) = tr \t(A - T T DT~ T ) T W 4 

■{A - T T DT~ T )T T + TCt T 

where 

A = K- 1 2(F + Pb k )K 1 2, W 4 = K2W 4 K2 

C = K2 (b k ) T (W 1 - P T W 4 P)b k K 1 2. 

From the foregoing arguments, the problem of obtaining matrices D and T that minimize (19) 
subject to the scaling constraints in (21) is now converted into an unconstrained optimization 
problem of obtaining D and t that jointly minimize /(D, T) in (25). 
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Let x be the column vector that collects the variables in matrix D and matrix \t\,tir • • ,t n ]> 
Then J(D, t) is a function of x, denoted by J(x). The proposed algorithm starts with an initial 
point xq obtained from an initial assignment D = f = I n . In the kth iteration, a quasi-Newton 
algorithm updates the most recent point x k to point x k+ i as [10] 



*fc+i = *k + oc k d k (26) 



where 



d k = -S k X7J(x k ) 

^k — ar S mm J( x k + ^d-k, 

h i = Sk + ( i + 7 * Sfc7 * ^ ^ - ^7?Sfc+Sfc7A r 



7*** y 7*A ll^k 

S = 1, S k = x M -x k , y k = V/(x fc+1 )-V/(x fc ). 

Here, V/(x) is the gradient of J(x) with respect to x, and S k is a positive-definite approxima- 
tion of the inverse Hessian matrix of }(x k ). This iteration process continues until 

l/(**+l) -/(**)!<* (27) 

is satisfied where e > is a prescribed tolerance. 

In what follows, we derive closed-form expressions oiVJ(x) for the cases where D assumes 

the form of a general, diagonal, or scalar matrix. 

1) Case 1: D Is a General Matrix: From (25), the optimal choice of D is given by 

D = t~ T At T , (28) 

which leads to 

J(t ~ T At T , t) = tr \tCt T ] . (29) 

In this case, the number of elements in vector x consisting of t is equal to n 2 and the gradient 
of J(x) is found to be 

3/(*) _ Hm J(fy)-J(fr) 



dtij A^O A (30) 

= 2ejtCT T tg ijr i,j = 1, 2, • • • , n 

where T,, is the matrix obtained from T with a perturbed (i,/')th component, which is given 
by 



Tij = T + 



i - *«; T gi j 



and g- ■ is computed using 



^ = 8 {|l|i} /a, V = ]^fl3^'/- INI 26 ')- 

2) Case 2: D Is a Diagonal Matrix: Here, matrix D assumes the form 

D = diag{di,d 2 , ••• ,d n }. (31) 
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In this case, (25) becomes 

]{D,t) = tx[tM d t T ] (32) 



where 



It follows that 



M d = C + A T W 4 A + W 4 T T D 2 T T 

-A T W 4 f T DT~ T - W 4 AT T DT~ T . 



dj(x) T ~ ~T- 

-^ = 2eJTM d T T 8ij , i,j = 1,2, • • • ,n 

dj(x 



(33) 



2eJ(DT -TA )W 4 f e it i = 1,2,- •• ,n. 



9^ 

3) Case 3: D Is a Scalar Matrix: It is assumed here that D = ocI n with a scalar oc. The gradient of 
]{x) can then be calculated as 

^ = 2ejtM s t T tg ijf i,j = 1,2, • • • ,n 

(34) 



4p^ = tr [t(2a:W4 - A T W 4 - W 4 A)t T 



where 

M s = (A - aI„) T W 4 (A - al„) + C. 

4. A NUMERICAL EXAMPLE 

In this section we illustrate the proposed method by considering a linear discrete-time system 
specified by 






1 
















1 


, b = 





0.339377 


-1.152652 


1.520167 




1 



A 

c = [ 0.093253 0.128620 0.314713 ] . 

Suppose that the poles of the observer and regulator in the system are required to be located 
at z = 0.1532, 0.2861, 0.1137, and z = 0.5067, 0.6023, 0.4331, respectively. This can be achieved 
by choosing 

k = [ 0.471552 -0.367158 3.062267 ] 

g = [ -0.006436 3.683651 5.083920 ] T . 

Performing the /2-norm dynamic-range scaling to the state-estimate feedback controller, we 
obtain /(0) = 686.4121 in (7) where D = 0. Next, the controller is transformed into the optimal 
realization that minimizes /(0) in (7) under the /2 -n o rm dynamic-range scaling constraints. 
This leads to / m7n (0) = 28.6187. Finally, EF and state-variable coordinate transformation are 
applied to the above optimal realization so as to jointly minimize the output roundoff noise. 
The profiles of J(x) during the first 20 iteration for the cases of D being a general, diagonal, 
and scalar matrix are depicted in Fig. 3. 
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1) Case 1: D Is a General Matrix: The quasi-Newton algorithm was applied to minimize (25). It 
took the algorithm 20 iterations to converge to the solution 



D 



0.211191 

-1.321589 

1.917916 

-11.039974 

-3.231505 

2.620911 



-3.078211 - 
1.897308 
-1.890027 - 

-43.683697 
8.919473 
6.462685 



-3.344596 

3.243515 

-3.807473 

-30.131793 
9.118205 
7.032260 



and the minimized noise gain was found to be J(D,T) = 4.8823. Next, the above optimal 
EF matrix D was rounded to a power-of-two representation with 3 bits after the binary point, 
which resulted in 

0.250 -3.125 -3.375 
-1.375 1.875 3.250 

1.875 -1.875 -3.750 



D 



3bit 



and a noise gain ](D^ it/ t) = 23.4873. Furthermore, when the optimal EF matrix D was 
rounded to the integer representation 



D 



int 



the noise gain was found to be J(D int , t) = 293.0187. 

2) Case 2: D Is a Diagonal Matrix: Again, the quasi-Newton algorithm was applied to minimize 
/(D, T) in (25) for a diagonal EF matrix D. It took the algorithm 20 iterations to converge to 
the solution 

D = diag{0.050638, -0.608845, -0.951572} 

3.588878 0.735966 0.010417 

-2.457241 0.728171 0.556762 

1.514232 -2.058856 0.142204 

and the minimized noise gain was found to be J(D,T) = 12.7097. Next, the above opti- 
mal diagonal EF matrix D was rounded to a power-of-two representation with 3 bits af- 
ter the binary point to yield D 3bit = diag{0. 000, —0.625, —1.000}, which leads to a noise 
gain J(D^j t/ t) = 12.7722. Furthermore, when the optimized diagonal EF matrix D was 
rounded to the integer representation Df nt = diag{0, —1,-1}, the noise gain was found to be 
J(D int ,T) = 13.7535. 

3) Case 3: D Is a Scalar Matrix: In this case, the quasi-Newton algorithm was applied to mini- 
mize (25) for D = &I3 with a scalar oc. The algorithm converges after 20 iterations to converge 
to the solution 

D = -0.779678 1 3 



3.252790 


-0.081745 


-0.198376 


1.717225 


1.220068 


-0.792487 


0.546599 


-0.854316 


2.295944 



and the minimized noise gain was found to be /(D, T) = 16.2006. Next, the EF matrix D = ocI 3 
was rounded to a power-of-two representation with 3 bits after the binary point as well as 
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Fig. 3. Profiles of iterative noise gain minimization. 



an integer representation. It was found that these representations were given by D^ it = 
diag{0.750, 0.750, 0.750} and Di nt = diag{l, 1,1}, respectively. The corresponding noise gains 
were obtained as J(D 3bit , t) = 16.2370 and J(D int , t) = 18.2063, respectively. 
The above simulation results in terms of noise gain /(D, T) in (25) are summarized in Table 1. 
For comparison purpose, their counterparts obtained using the method in [9] are also included 
in the table, where the minimization of the roundoff noise was carried out using EF and state- 
variable coordinate transformation, but in a separate manner. From the table, it is observed 
that the proposed joint optimization offers improved reduction in roundoff noise gain for the 
cases of a scalar EF matrix and a diagonal EF matrix when compared with those obtained by 
using separate optimization. However, in the case of a general EF matrix, the optimal solution 
with infinite precision appears to be quite sensitive to the parameter perturbations. 



Error-Feedback 


Accuracy of D 


Scheme 


Infinite 
Precision 


3 Bit 
Quantization 


Integer 
Quantization 


D = 

Separate 


28.6187 


Scalar 
Separate [9] 


20.1235 


20.1810 


26.0527 


Scalar 
Joint 


16.2006 


16.2370 


18.2063 


Diagonal 
Separate [9] 


16.4104 


16.4547 


17.4039 


Diagonal 
Joint 


12.7097 


12.7722 


13.7535 


General 
Separate [9] 


11.6352 


11.7054 


16.5814 


General 
Joint 


4.8823 


23.4873 


293.0187 



Table 1. Noise gain J(D,T) for different EF schemes. 



More reduction of the noise gain might be possible by re-designing the coordinate transfor- 
mation matrix T for the optimally quantized D. 
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5. CONCLUSION 

The joint optimization problem of EF and realization to minimize the effects of roundoff 
noise of the closed-loop system with a state-estimate feedback controller subject to /2-norm 
dynamic-range scaling constraints has been investigated. The probelm at hand has been con- 
verted into an unconstrained optimization problem by using linear algebraic techniques. An 
efficient quasi-Newton algorithm has been employed to solve the unconstrained optimization 
problem. The proposed technique has been applied to the cases where EF matrix is a general, 
diagonal, or scalar matrix. The effectiveness for the cases of a scalar EF matrix and a diag- 
onal EF matrix compared with the existing method [9] has been illustrated by a numerical 
example. 
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1. Introduction 

Many endogenous and exogenous factors can affect the physiological, mental and 
behavioral states in humans. In order to identify such states, monitoring tools need to use 
biological indicators, or biomarkers, able to identify biological events and predict outcomes. 
These biomarkers can be divided into two categories. 

The first category contains what we could call the "structural" biomarkers that are extracted 
from physiological structures and mainly defined at the genetic and/ or molecular level (e.g., 
Berg, 2008; Dengler et al, 2007; Eleuteri et al, 2009; Isaac, 2008; Moura et al, 2008; Wei, 
2009). For instance, the formation or consumption of certain molecules provide biomarkers 
to identify patients with moderate to severe forms of cardiac heart failure (Eleuteri et al., 
2009; Isaac, 2008) while changes in Cortisol level allow detection of an increased stress 
response (Armstrong & Hatfield, 2006). Similarly, other active molecules (e.g., C-reactive 
protein) are used as biomarkers of valvular heart disease (Moura et al., 2008) while cardiac 
troponins and N-type natriuretic peptides can be used in post-transplant patient 
surveillance (Dengler et al., 2007). Other examples of structural biomarkers aim to identify 
abnormalities in neural connectivity in the brain. For instance, the presence of certain 
molecules in venous blood or a damaged white matter provides potential predictors of risk 
of cerebral palsy (Dammann & Leviton, 2004, 2006; Kaukola et al., 2004). Also, genomic and 
proteomic biomarkers are able to define the risk of an individual to develop a 
neurodegenerative disease such as Parkinson's disease (Gasser, 2009), Alzheimer's disease 
(Berg, 2008; Wei, 2009) or amyotrophic lateral (Tuner et al., 2009) and multiple sclerosis 
(Wei, 2009). 

The second category includes what we could call "functional" biomarkers that are further 
related to continuous measurements of body function throughout time in order to track 
physiological, mental and behavioral states (e.g., Georgopoulos et al., 2007; Hejjel & Gal, 
2001; Hofstra et al., 2008). For instance, electro-cardiograms, heartbeat, and body 
temperature are possible functional biomarkers to determine stress level (Hejjel & Gal, 
2001). Body temperature can be used to detect the phase of circadian rhythms (Hofstra et al., 
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2008), and blood pressure can be employed to identify the chronic fatigue (Newton et al., 
2009). Recently, it has been also suggested that measurements of the skin conductance was a 
better tool to monitor nociceptive stimulation and pain than heart rate and blood pressure 
(Storm et al, 2008). 

Another important family of functional biomarkers includes status measurements of brain 
functions in order to monitor and interpret neural activity, identify specific neurological 
events and predict outcomes (e.g., Gentili et al., 2008; Guarracino, 2008; Hatfield et al., 2004; 
Irani et al, 2007; Tuner et al, 2009; van Putten et al., 2005; Williams & Ramamoorthy, 2007). 
These brain indicators, or brain biomarkers, can be derived from signals recorded by means 
of invasive acquisition techniques such as implantable microelectrodes arrays or 
electrocorticography (Schalk et al., 2008), or, alternatively, non-invasive techniques such as 
electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic 
resonance imaging (fMRI) or emerging neuroimaging technologies such as functional near 
infrared spectroscopy (fNIRS) (Irani et al., 2007; Parasuraman & Rizzo, 2007). For instance, 
brain biomarkers derived from temporal or spectral EEG signals processing allow for the 
determination of anesthetic depth during pediatric cardiac surgery (Williams & 
Ramamoorthy, 2009). Other brain biomarkers derived from EEG, such as the brain 
symmetry index, permit the detection of seizure activity in the temporal lobe and can be, 
therefore, useful for epileptic monitoring needed in intensive care units (van Putten et al., 
2005). Still using EEG analysis, it is also possible to detect a reduction of cerebral blood flow 
below a certain threshold (Guarracino et al., 2008). Other high-temporal resolution 
measurement techniques such as MEG have also been used to successfully classify 
respective groups of individuals subjected to multiple sclerosis, Alzheimer's disease, 
schizophrenia, Sjogren's syndrome, chronic alcoholism, facial pain and healthy controls 
(Georgopoulos et al., 2007). More recently, it has been shown that the fNIRS imaging 
technique, a relatively novel cerebral imaging tool, could provide information allowing the 
monitoring of brain oxygenation by measuring regional cerebral venous oxygen saturation 
(Guarracino et al, 2008). 

These examples provided by medical, biomedical and bioengineering research fields 
illustrate how various brain monitoring tools are being developed intending to uncover 
structural or functional brain biomarkers for detection, prevention, prediction, and 
diagnosis of heart function, adverse neurological events and neural/ neurodegenerative 
diseases. However, the research aiming to uncover functional brain biomarkers directly 
relevant for the restoration of cognitive-motor and/ or sensorimotor functions (e.g., disabled 
populations, advanced aging) is still a relatively young research field. Indeed, although 
many assistive technologies aiming to restore cognitive-motor and sensorimotor functions 
are currently underway (e.g., neuroprosthetics (Cipriani et al., 2008; Wolpaw et al., 2007); 
exoskeletons (Carignan et al., 2008)), few brain monitoring tools related to sensorimotor 
integration are being developed. However, these bioengineering applications, such as the 
design of smart neuroprosthetics, require a deeper understanding of brain dynamics in 
ecological situations that involve human interaction with new tools and/ or changing 
environments that guide learning and more generally shape motor behavior. Specifically, 
such monitoring tools aiming to assess the dynamic status of the brain necessitates the 
knowledge of brain biomarkers able to track brain dynamics in ecological situations where 
humans have to learn new tasks, to master novel tools and/ or changing environments. 
These brain biomarkers should be preferably non-invasive (i.e., no surgical intervention 
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needed), simple to record and analyze, simultaneously robust and sensitive to specific 
changes in brain function in natural situations. Such assessment in ecological situations 
requires non-invasive recording of the dynamic brain activity with a high temporal 
resolution (e.g., millisecond), which is well suited for EEC Although some research efforts 
are underway (e.g., Deeny et al, 2003, 2009; Gentili et al, 2008, 2009a,b; Hatfield et al, 2004; 
Haufler et al., 2000; Kerick et al., 2004) to develop methods to provide non-invasive 
functional brain biomarkers able to track the brain status during sensorimotor performance; 
some questions and problems remain. For example, how accurately and efficiently can a 
cognitive-motor or sensorimotor state be inferred? What methods might provide robust 
brain biomarkers applicable on single-subject and single-trial bases? How can the signal 
processing techniques used in laboratory contexts to derive such biomarkers can be 
transferred successfully in real-time applications to ecological contexts? Although this 
manuscript does not purport to exhaustively answer these questions, some elements of 
response and possible problem-solving perspectives will be presented and discussed. 
Therefore, the aims of this chapter are to provide the state-of-the-art of the research along 
with the main signal processing techniques related to functional non-invasive EEG/MEG 
brain biomarkers that allow tracking of cortical dynamics to assess the level of mastery of a 
sensorimotor task and the adaptation to novel tools or environments. It must be noted that, 
from a technical point of view, the methodological approaches presented here are also 
applicable to some (minimally) invasive techniques such as electrocorticography. However, 
when considering an invasive approach, in addition to the inherent risks and difficulties 
related to a surgical intervention, the whole scalp will not be likely covered by the recording 
device, creating limitations in terms of the regions of interest where potential biomarkers 
could be detected. Thus, we will mainly focus on non-invasive recording techniques that use 
a high- temporal resolution (EEG/MEG) with a particular emphasis on results obtained with 
EEG since this recording technique is portable and, thus, applicable in ecological situations. 
In Section 2, the main pre-processing methods employed to clean the EEG/MEG signals of 
artifacts will be explained along with the subsequent methodological approaches that allow 
for the computation of brain biomarkers. Specifically, Section 2 will focus on the spectral 
power and phase synchronization representing the two most classical univariate and 
multivariate non-invasive functional brain biomarkers of performance. In Section 3, the 
classical and the latest findings in this brain biomarker research field will be presented by 
emphasizing promising progress but also current limitations and possible solutions to 
overcome them. Section 4, will present how these brain biomarkers may provide important 
advances in bioengineering applications in ecological contexts such as the development of 
smart neuroprosthetics and brain monitoring techniques. Finally, we will summarize these 
results and suggest future research directions. 

2. Signal Processing Methods 

The aim of this second section is not to provide an exhaustive presentation of all the existing 
processing methods for EEG and MEG signals, but rather, to introduce some signal 
processing approaches for EEG and MEG signals to, first, pre-process the signal to remove 
artifacts and, then, to derive non-invasive functional brain biomarkers (e.g., based on 
spectral power and coherence) that are used to assess and track adaptation in cognitive- 
motor/ sensorimotor performance in humans. 
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2.1 Pre-processing 

During recording, EEG/MEG signals are generally corrupted with some undesirable 
artifacts such as body movements, muscular artifacts, eye movements, eye blinks, 
environmental noise or heart beat. These artifacts produce possible biases in the detection 
and interpretation of brain biomarkers that will be later derived from the EEG/MEG 
signals. Constraints placed on subjects to minimize these artifacts in a laboratory setting 
cannot be realistically expected in an ecological situation. Therefore, in order to remove such 
artifacts, pre-processing of the EEG/MEG signals may be a necessary and critical step 
(Georgopoulos et al., 2007). Although several signal processing methods are available, such 
pre-processing stage can be performed by using various methods such as Independent 
Component Analysis (ICA) and adaptive filtering. 

2.1.1 Artifact removal using Independent Component Analysis 

In many dynamical systems, the measurements are given as a set of mixed signals with 
noise. For example, in the same way conversations are recorded by a number of 
microphones in a crowded party, brain signals containing artifacts are measured through 
multiple EEG/MEG sensors. The information in each of the original signals can be analyzed 
as long as it is possible to identify the system corresponding to the source that emits these 
signals captured by a set of sensors. In this regard, blind source separation is a relevant 
method to approximately recover the original source signals from a set of observed mixed 
signals without any a priori knowledge about either the source signals or the mixing system. 
Regarding applications in biomedical signal processing, ICA is currently considered one of 
the most sophisticated statistical approaches for solving the general problem of blind source 
separation. 

2.1.1.1 Basic assumptions of ICA 

ICA is a linear transformation method to find estimated source signals (i.e., the independent 
components) while optimally demixing the mixed signals where independent components 
must satisfy the following conditions (Hyvarinen & Oja, 2000; Oja, 2004; Vaseghi, 2007; 
Vigario et al, 2000): 

i) The independent components are non-Gaussian and statistically independent of 

the higher-order statistics (covariance and kurtosis). 
ii) At most, no more than one independent component can be Gaussian, 
iii) The dimension of the set of independent components does not exceed the number 
of sensors. 

Moreover, three additional assumptions must be considered when ICA is applied to 

EEG/MEG signals (Hyvarinen et al, 2001): 

iv) The existence of statistically independent components in EEG/MEG source signals 

is assumed, 
v) The statistically independent components are instantaneously and linearly mixed 

at the sensors, 
vi) The independent components and the mixing processes are supposed to be 

stationary. 
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Several versions of ICA exist. First, the simple ICA will be presented. Then, the two most 
popular ICA algorithms named Infomax ICA and FastICA will be reviewed. 

2.1.1.2 Simple ICA algorithm 

In the simple ICA algorithm, the unknown additive noise is excluded (Oja, 2004). Assume 
that the TYl dimensional observed signal (e.g., EEG/MEG) vector 

X\k ) = [X l \k), X 2 \k), •••, ^ m (^)J is given by a linear combination of the fl 

dimensional source signal vector S[k ) = \S l (k J, S 2 [k), "", ^«(A/J a * each time 
sample k , that is: 

xXk) = a il s l (k)+a i2 s 2 (k)+-'- + a in s n (k)j = l,2,---,m. (l) 



In a more compact notation, Equation (1) can be rewritten as 

n 
7=1 

where the matrix A = |a 19 a 2 , ••-, a w J is the mixing matrix, the indices n and m 

are the number of sensors and sources, respectively. The matrix A is a m x n matrix 
(generally m > n but a common choice is m = n). Practically, both the mixing matrix and the 
source signal vector are unknown; however, we can estimate a demixing matrix W in 
order to obtain the estimation of a source signal vector S\k) using three fundamental 
assumptions (from i) to iii); see section 2.1.1.1) for ICA previously mentioned such that: 

s(k)=Wx(k) (3) 

where ideally W = A and the elements of S\k) are statistically independent. 

Practically, several preprocessing strategies make ICA simpler and better conditioned 
(Hyvarinen & Oja, 2000). For example, the centering technique simplifies the ICA algorithms 
by subtracting the mean vector from the observed signal vector so as to make it a zero mean 
valued vector. On the other hand, whitening decreases the correlation among the observed 
signals by transforming the centered observed vector to have unit variance in all directions 
(Vigario, 2000). 

2.1.1.3 Infomax ICA and FastICA 

Among the various ICA algorithms that are available, Infomax ICA (Bell & Sejnowski, 1995) 
and FastICA (Hyvarinen, 1999) are the two most popular ones. They use different 
independence properties to obtain the independent components. Specifically, Infomax ICA 
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minimizes the mutual information whereas FastICA maximizes the non-Gaussian nature. 
These two algorithms provide qualitatively and quantitatively similar results. However, 
FastICA is generally faster than Infomax ICA, but is subject to more variability than Infomax 
ICA especially when applied to removal of eye blink artifacts (Glass et al., 2004). Concerning 
Infomax ICA, this approach is unable to separate source signals with a sub-Gaussian 
distribution. Therefore, an extended version of Infomax ICA, named extended Infomax ICA, 
has been introduced to separate both sub-Gaussian and super-Gaussian distributions for the 
source signals (Lee et al., 1999). 

2.1.1.4 Independent Components Analysis for artifact identification and removal from 
EEG and MEG signals 

ICA has been recently applied to the analysis of biomedical signals mostly acquired from 
EEG and MEG. In these applications, it is essential to associate each independent component 
with the neurophysiological nature of the phenomenon (e.g., event-related brain dynamics, 
steady-state brain activity, etc.) in order to identify them. In many cases, ICA algorithms 
have been successfully applied to EEG and MEG in order to identify and remove artifacts 
such as cardiac, ocular, or muscular activities from the neurophysiological activities of 
interest (the computational steps of these algorithms are illustrated in Fig.l), since the nature 
of the artifact sources is different from those of the actual brain activity related sources in 
terms of anatomical, physiological, and statistical considerations. 



Biomedical signal 
(EEG / MEG) 



Optional Pre-processing 
( Ce nte r i ng, D e c i mati o n , etc . 



ICA 
(Infomax ICA, FastICA, etc. 



A rtif a ct i d e ntif i c ati o n a n d r e mo va I 
(Manually or Automatically) 



Analysis of artifact-cleaned biomedical signal 
(Spatial, spectral, and temporal analysis) 



Fig. 1. Computational steps for ICA-based signal processing. 



In general, the independent components related to the suspected artifacts must be manually 
assigned to an artifact type based on the attributes of the independent components (e.g., 
amplitude peak, frequency patterns). However, since the criteria to decide to remove such a 
component can depend of subjective judgments, this approach is sensitive to biases. 
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Recently, several automatic artifact detection and removal methods have been introduced 
(Delorme et al., 2001; Rong & Contreras-Vidal, 2006). For example, the functionally similar 
independent components could be automatically categorized using neural network with 
respect to a set of features such as spatial maps, spectral properties, and higher-order 
statistics (Rong & Contreras-Vidal, 2006). 

2.1.1.5 Limitation of ICA 

Although ICA facilitates the analysis of the brain dynamics, this method cannot isolate 
highly correlated sources due to the assumption of statistical independence. Furthermore, it 
cannot identify uniquely ordered, correctly phased and properly scaled source signals, in 
other words, when using ICA, the independent components that are isolated could be 
randomly ordered, reversely phased, or ill scaled. However, in the case where such specific 
characteristics are of interest, it must be noted that ICA is not able to identify the source of 
the signals. Moreover, for practical bioengineering applications, artifact identification and 
removal based on ICA is not appropriate for real-time processing since it requires significant 
computational resources and a large amount of data collected from a sufficiently large 
number of channels. The next paragraph introduces adaptive filtering, another method that 
can be potentially useful for real-time applications. 

2.1.2 Artifact removal using adaptive filtering 

Despite the advantages of ICA as an artifact removal method, this technique is 
computationally very expensive and, thus, not well suited under some conditions such as 
real-time applications. However, other linear and nonlinear filtering based-techniques to 
remove specific artifacts in real-time are available. Among these methods, adaptive filtering 
has been introduced for removing ocular artifacts in real-time (He et al., 2004). 

2.1.2.1 Principle of adaptive filtering 

Adaptive filters are based on the principle that the desired (clean) signal can be extracted 
from the input signal through the adaptation of the filter parameters. The filter parameters 
are adapted based on minimizing an error function between the filter output signal and a 
desired signal. The most commonly used adaptive filtering algorithms are the Kalman filter, 
the least mean square (LMS) filter, and the recursive least square (RLS) filter (for more 
details on the implementations of these methods see Zaknich, 2005). 

2.1.2.2 Removing ocular artifacts by adaptive filtering 

Specifically, adaptive filtering has been used to remove ocular artifacts that could 
contaminate EEG/MEG (Georgiadis et al., 2005; Sanei & Chambers, 2007). For instance, He 
et al., (2004) suggested an adaptive filter that uses three inputs to the system. First, the actual 
EEG/MEG signal x(k) with the ocular artifacts z(k) as the primary input 

( S\k) = x(k)+ z(k) ). The second and third inputs are the vertical and horizontal eye 
movement (VEOG and HEOG) as two reference inputs (r v [k) and r h (&)), respectively. Each 
reference input is first processed by a finite impulse response (FIR) filter using the RLS 
algorithm (r v [k) and f h {k), respectively) and then subtracted from the EEG signal under 
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the assumption that the desired ocular artifacts cleaned EEG signal is a zero-mean stationary 
random signal that is uncorrelated with the ocular artifacts and the two reference signals. 
Thus, the desired output produced by the whole system is the EEG signal without ocular 
artifacts. Hence the whole system can be described using the following sets of equations and 
the corresponding scheme illustrated in Fig. 2: 



e{k) = s{k)-r v {k)-r h {k) = x{k) + [z{k)-r v {k)-r h {k)] 



(4) 



where r v (k) = ^h v (m)r v (k + 1 -m) and r h {k) = ^h h (m)r h (k + 1 -m) for the filter 



777=1 



parameters h v [pi) and h h [pi), respectively. e\k) is the error between the observed signal 
and reference inputs. 



lnput prlrnary (EEG) 

r v (k) 



lnput r=fl (VEOG) 



r & {k) 



lnput raf2 (HEOG) 



S'k' = X'k'+Z'k' 



Adaptive filter 
H Am) 



RLSfilter 

h y \m I 



rJk) 



_i 

RLSfilter 
h k \m\ 



ei/t'i 



f,(t) 



-»o 



Output [cleaned EEG) 



Fig. 2. Computational scheme of the adaptive filter configuration for eye artifact removal 
(EOG: Electrooculography). (The different symbols used in this figure are described in the 
text above). 

2.1.2.3 Limitation of adaptive filtering 

The LMS and RLS filters, popular alternative algorithms to the Kalman filter, also present 
some advantages and drawbacks. The LMS filter is one of the relatively simple adaptive 
filtering algorithms, so it is computationally very efficient, but it is not suitable for signals 
with high rate of sudden changes due to its slow rate of convergence (Vaseghi, 2007). In this 
case, the RLS filter offers relatively faster convergence and smaller error rate with more 
computations. More generally, a single type of artifact can be removed with a single filter, so 
multiple filtering must be performed when multiple forms of artifacts are present, 
increasing the chances of distorting the signals of interest. 



2.1.3 Summary 

To summarize, many novel artifact-removal techniques have been introduced along with 
some of their variants (He et al, 2004; Lee et al, 1999; Vaseghi 2007; Vigario et al, 2000). A 
common requirement for the artifact removal method is to remove the artifacts but keep the 
neurophysiological activities of interest intact. For this reason, the employment of 
algorithms for modeling and filtering must be carefully considered along with their 
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underlying assumptions, since it may undesirably alter the estimated artifact-cleaned 
EEG/MEG signal (Hyvarinen & Oja, 2000; Oja, 2004; Vaseghi, 2007; Vigario et al, 2000). ICA 
is generally the most suitable artifacts removal algorithm with minimal affects on the 
interesting EEG/MEG signals, but it is very expensive in terms of both computation and 
memory usage. Adaptive filtering, on the other hand, can effectively remove artifacts from 
EEG/MEG signals in real-time fashion. 

Once EEG/MEG signals are free of artifacts, the next step is to compute the brain 
biomarkers derived from these clean EEG/MEG signals in order to assess sensorimotor 
performance and learning. In this regard, the two main biomarkers that are available are 
derived from the spectral power and phase synchronisation between two signals located at 
different positions on the scalp. These two brain biomarkers are presented in the next two 
sections. 



2.2 Spectral Power 

A first type of brain biomarker that can be used to assess the level of mastery in 
sensorimotor performance and learning can be derived from the spectral power computed 
for specific frequency bands. Many different methods (e.g., parametric, non-parametric, and 
subspace methods) are available to compute the EEG/MEG spectral power (Kay, 1988; Sanei 
& Chambers, 2007; Shumway & Stoffer, 2000). For instance, some of these methods that have 
been applied are the classical fast Fourier transform (e.g., Hatfield et al., 1984; Haufler et al., 
2000) and more sophisticated procedures such as the multitaper (e.g., Conteras-Vidal & 
Kerick, 2004) or wavelet (e.g., Mu et al., 2008) techniques. While some of these approaches 
have been applied with success in EEG/MEG studies that focus on sensorimotor 
performance and/ or Brain Computer Interface (BCI) systems (McFarland et al., 2006; 
Pfurtscheller & Lopes da Silva, 1999), two methods are particularly popular to compute the 
EEG/MEG spectral power. The first approach uses autoregressive (AR) methods (e.g., 
McFarland et al., 2006, 2008) while the second one uses the band power method 
(Pfurtscheller & Lopes da Silva, 1999, 2005; Pfurtscheller & Neuper, 2006) providing changes 
in power amplitude that are often referred to as " event related desynchronization (ERD)" 
and " event related synchronization (ERS)." 

2.2.1 Autoregressive filtering 

The first technique that consists of using AR models is a classical parametric method 
(Marple, 1987; Sanei & Chambers, 2007; Shumway & Stoffer, 2000). Contrary to the fast 
Fourier transform, parametric spectral estimation by means of AR models offers various 
advantages by presenting a more general and flexible framework for parsimonious 
dynamical modeling of time series data useful for different applications such as prediction, 
classification or causality analysis of time series (Shumway & Stoffer, 2000; Wong et al., 
2006). Specifically, an AR filter can be used for linear prediction in order to model the signal 
of interest; here an EEG or MEG signal. Namely, the real EEG/MEG signal can be 
considered as the sum of the signal modeled by the AR filter and an error term. Thus, by 
subtracting the real EEG/MEG signal to the one filtered by the AR model, the prediction 
error can be determined (Fig.3). 
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Fig. 3. Principle of linear prediction using an AR filter. (The different symbols used in this 
figure are described in the text below). 

The prediction error for an AR model is defined as: 

r=p 

e(k) = x(k) - ^T a r x(k - r) 



r=\ 



(5) 



where a r (r = 1,2,3,... ,p) are the coefficients, the constant p is the order of the filter, and k 

denotes the discrete time sample. x(k) and e(fc) dace respectively the input signal to 
approximate and the prediction error. For a given p, the coefficients are identified by 
minimization (e.g., LMS, Durbin method) of the error or driving signal that is considered to 
be zero mean white noise. 

By applying the z-transform to equation (5) and considering Z=Q J0) we obtain: 



X{&) 

E(co) 



1 



r=p 



(6) 



i-2X< 



-jrco 



r=\ 



Where E(ti)) represents the power spectrum of the white noise that is constant (i.e., 
E(co) = K^ ), and X(co) represents the power spectrum of the signal. From this model, the 

spectral power can be estimated for any specific frequency band. 

AR models may suffer from poor estimation of the model parameters mainly due to the 
limited length of the measured signal (Sanei & Chambers, 2007) while the order of the AR 
filter may influence the precision of the computation of power spectrum. For instance, 
McFarland et al., (2008) recently showed that the resolution of lower frequency signals 
requires higher AR model orders and also that increasing AR model order provided an 
enhanced spectral resolution. It must be noted that an increase of the AR model order 
results in a higher computational cost even if the tremendous advances in digital signal 
processor and field-programmable gate-array technology tend to weaken this drawback 
(Wang et al., 2006). Also, in the case of non-stationarity, parametric spectral estimation may 
also be applied with a moving window (Ozaki & Tong, 1975) or using some alternative 
approaches avoiding, thus, the introduction of such a moving window (Wong et al., 2006). 
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2.2.2 ERD/ERS method 

The second method is well-established and has been successfully applied to many different 
EEG/MEG investigations (Gentili et al, 2008 ,2009a, 2009a; Kerick et al, 2004; Pfurtscheller 
& Lopes da Silva, 1999, 2005; Pfurtcheller & Neuper 2006; Tombini et al, 2009). Specifically, 
this method computes the spectral power by squaring and averaging across trials the output 
of a band pass filter in order to detect the changes in power amplitude. ERD and ERS 
correspond respectively to a decrease and an increase of the spectral power for specific 
frequency bands (e.g., alpha band) and brain regions (e.g., frontal region). These measures 
are often expressed as a percentage of a decrease or increase with respect to a baseline 
condition preceding task performance and are computed according to the following 
equation (for more details see Pfurtscheller & Lopes da Silva, 1999): 

ERS I ERD = Pe ~ Pr xl00 (%) (7) 

where P E and P R correspond to the power computed within the frequency band of interest 

in the period after the event begins and during the preceding baseline or reference period, 
respectively. 

It must be noted that although these ERD/ERS quantifications can be computed using 
different methods including AR filters, (e.g., see Table 1 in Pfurtscheller & Lopes da Silva, 
1999) the term ERD/ERS is generally associated with the band pass method (see 
Pfurtscheller & Lopes da Silva, 1999, 2005 for a comprehensive review). From a 
physiological point of view, ERD/ERS mirror variations of the activity of local interactions 
between main neurons and interneurons that control the frequency components of the 
ongoing EEG (Pfurtscheller & Lopes da Silva 1999, 2005). As previously mentioned, 
although several methods can be used to isolate some specific frequency bands; one of the 
main problems of the EEG/MEG spectral analysis is the definition of the upper and lower 
bounds of the bands (Pfurtscheller & Lopes da Silva, 1999). Although the definition of the 
frequency band limits can slightly differ from one study to another, a possible approach for 
partitioning the frequency bands related to human motor performance for healthy adults is 
to consider the theta ([4-7 Hz]), alpha ([8-13 Hz]), beta ([14-35 Hz]) and gamma ([36-44 Hz]) 
frequency bands (e.g., Hatfield et al, 2004; Haufler et al, 2000; Tombini et al, 2009). 
Sometimes, the frequency range spread from 8 to 15 Hz (Blanker tz & Vidaurre, 2009) or 
from 9 to 13Hz (Blankertz et al., 2009; Pfurtscheller & Neuper, 1997) are also named alpha 
frequency (or mu rhythm under certain conditions). Moreover, since it has been shown that 
certain frequency sub-bands are related to specific brain states during a sensorimotor task 
(e.g., Contreras-Vidal et al, 2004; Gentili et al., 2008; Hatfield et al, 2004; Tombini et al, 
2009), most of the EEG/MEG studies refined their analysis by considering sub-frequency 
bands, typically, the low and high component of the original entire band. Therefore, for the 
bands previously defined, the low theta ([4-5 Hz]), high theta ([6-7 Hz]), low alpha ([8-10 
Hz]), high alpha ([11-13 Hz]), low beta ([14-23 Hz]) and high beta ([24-35 Hz]) frequency 
bands can also be considered. In addition to the classical gamma band ([36-44 Hz]) it is also 
possible to consider the extended gamma band spread from 45 to 100 Hz or higher. This 
gamma band extension can be divided into several sub-bands with a method using a 10-Hz- 
wide band with an overlap of 5 Hz frequency bins ranging from 45 to 100 Hz (Crone et al., 
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1998). Although, as previously mentioned, the limits of these bands can slightly change from 
one study to another; many EEG/MEG investigations consider frequency bands where 
upper and lower limits of the bandpass filter is the same for all the subjects tested. It must be 
noted that another approach (Pfurtscheller & Lopes da Silva, 1999, 2005) defines these 
frequency band limits for each individual subject in order to take into account some inter- 
individual differences. For instance, three possible methods can be used to determine the 
upper and lower limits of the bandpass filter; i) detection of the most reactive frequency 
band by comparing the two short-term power spectra; ii) use of a continuous wavelet 
transform; iii) definition of frequency bands relative to the spectral peak frequency (for more 
details see Pfurtscheller & Lopes da Silva, 1999, 2005). 

2.3 Phase synchronization: Coherence and Phase Locking Value 

Another important brain biomarker of sensorimotor performance can also be provided by 
analyzing the phase synchronization between different cortical sites. Such phase 
synchronization measures the level of interaction and cross talk among EEG/MEG channels 
allowing the identification of how signals propagate within the neural network of the brain. 
These spatial EEG/MEG coherence measures, generally presented for individual frequency 
bands, result from correlations among different cortical sources. Therefore, spectral 
coherence is a common method for determining synchrony in EEG/MEG activity. 
Regarding the literature aiming to find brain biomarkers for human sensorimotor 
performance and learning, spectral power analysis has been widely used for a long time, 
however, the use of spectral coherence is relatively more recent, while the phase locking 
value (PLV), despite its advantages, still remains rarely used in this field. Generally, the 
literature focusing on EEG/MEG signal analysis computes the synchronization between two 
time signals recorded from two electrodes x and y by using classical coherence (Nunez & 
Srinivasan, 2006). First the cross-spectrum (CS) has to be computed using the following 
equation: 



xy * 



CS(f)= S x (f)Sy(f) (8) 



where S (/") is the Fourier transform of the signal s x (t) , S x (/") is the complex conjugate 

of the Fourier transform of the signal s x (t) and ( ) is the expectation operator. Then, the 

complex coherence (CC) is computed by using the cross-spectrum normalized with respect 
to the two corresponding spectra of the two signals. Thus we have: 

CS ( f) 

CC xy (/) = (9) 

JCSJf)CS yy (f) 

Where CS X (f) is the cross-spectrum of the two time signals s x (t) and s (t) and 
CC x (/") the complex coherence. 
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Finally, the coherence (C) can be calculated by considering the absolute value of the complex 
coherence: 

C xy (f) = \cC xy (f)\ (10) 

Another way to interpret these equations is to consider the following equation: 

S x (f) = P x e M (ii) 

where the Fourier transform S(f) of the signal s x (t) is expressed in order to explicitly 
illustrate its amplitude p x and its phase (p (here; denotes the imaginary unit and; 2 =-l) . 

Now the cross-spectrum expressed in equation (8) can be rewritten as: 

CS xy (f) = (p xPy e^) (12) 

where Aq? denotes the phase difference between the two signals (i.e., A(p = (p — (p ). 
Thus, the complex coherence expressed in equation (9) can be rewritten as: 

cc m k££l 

CC xyU)-— .. 2 , (13) 



Leading to the classical coherence provided by the following equation: 



CM) 



p,p/ a ") 



p 2 M 



(14) 



Although this measure of classical coherence is usually used in EEG/MEG studies, two 
main drawbacks have to be considered (Lachaux et al., 1999). First, the coherence can be 
applied only to stationary signals. Most of the time this assumption of stationarity (in time 
or across trials) is not strictly valid, however, the measure of phase-locking does not require 
this assumption on the signal. Second, coherence does not specifically quantify phase 
relationships. In fact, coherence increases with amplitude covariance (see the presence of the 

signal amplitudes p and p in the numerator and denominator of the formula in equation 

(14)) implying an uncertainty concerning the relative importance of amplitude and phase 
covariance in the coherence. In other words, the coherence does not separate the effects of 
amplitude and phase in the interrelations between two signals. Thus, based on these 
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premises and since phase-locking provides a measure that is sufficient to conclude if two 
brain regions interact, another measure of phase synchronization, the PLV, has been 
introduced, offering, thus, an alternative measure only based on the detection of phase 
covariance (Lachaux et al., 1999; Le Van Quyen et al., 2001; Tass et al., 1998). 
Before computing the PLV, the frequency bands and sub-bands of interest mentioned in 
Section 2.2.2 are extracted for each subject and each single-trial by means of a filter bank 
using band-pass FIR (Lachaux et al., 1999) or IIR filters (Brunner et al., 2006). 

Then, the PLV can be computed for each frequency band. Contrary to the classical 
coherence, it is defined by only considering the phases of the two signals. 



PLV = 



e jA,p ) 



(15) 



where A(p denotes the phase difference between the two signals s x (t) and 5 (t) (i.e., A(p 

It must be noted that equations (14) and (15) are comparable; however, the equation 
expressing the PLV does not include the amplitudes of the two signals, allowing 
examination of synchronization phenomena in EEG/MEG signals by directly capturing the 
phase synchronization. 

Two methods to compute the phases (p x and (p are available. The first one (Lachaux et al., 
1999) uses a complex Gabor wavelet as defined by equation (16): 



G(tJ) = e a e 3im (16) 



t 2 



Where a= — , t represents the time and (J is the standard deviation of the Gaussian 

2cr t 
envelope. 

The second method (Tass et al., 1998) uses the Hilbert transform as defined by the following 
equation: 



s * 



(t) = -Pvl^XiT (17) 

n J t - t 



In this definition, ^ x (t) is the Hilbert transform of the time series S x (t) (in our case an 

EEG/MEG signal), and PV indicates that the integral is taken in the sense of Cauchy 
principal value. The instantaneous phase can then be calculated as: 

5 (t) 

P*(0 = arctan^— ( i 8 ) 



Signal processing for non-invasive brain 

biomarkers of sensorimotor performance and brain monitoring 475 

It must be noted that these two methods provide very similar results when applied to EEG 

data (Le Van Quyen et al, 2001). 

The averaging process can be performed either over time (i.e., in equation (19), n E [1...N], 

where n is the sample number of the time series) for single-trial applications such as BCI 

approaches (Brunner et al., 2006; Lachaux et al., 2000) or over trials (Lachaux et al., 1999) 

(i.e., in equation (19), n E [1...N], where n is the trial number). Thus, equation (19) is 

obtained: 



PLV = — 

N 



N 

z 

1=1 



jAtp(t,n) 



(19) 



where A(p(t, ft) is the phase difference and A(p{t, n) = <p x (t, n) - (p (t, n) . 

As for the coherence, the PLV is ranged from to 1 indicating that during this time window 
the two channels considered are ranged from unsynchronized to perfectly synchronized, 
respectively. It must be noted that, despite the previously mentioned advantages of the PLV, 
it has been also suggested that one reason to use coherence rather than the PLV directly is 
that coherence measures are weighted in favor of epochs with large amplitudes. In 
particular, more consistent phase estimates will be probably obtained when amplitudes are 
large (if large amplitudes show a large signal-to-noise ratio as is generally the case in 
EEG/ MEG) (Nunez & Srinivasan, 2006). Therefore, both coherence and PLV measures can 
be used. Interestingly, due to their unique advantages and pitfalls, some studies apply and 
compare both techniques that, in the case of convergence, lead to robust results, although in 
the case of EEG both approaches are subject to the electrode reference problem that can 
distort such measurements (Nunez & Srinivasan, 2006). Recently, Darvas et al., (2009) have 
proposed an extension of the PLV, called bi-PLV that is specifically sensitive to non-linear 
interactions providing, thus, robustness behavior to spurious synchronization arising from 
linear crosstalk. This property is particularly useful when analyzing data recorded by EEG 
or MEG. From a physiological point of view, both coherence and PLV methods quantify the 
magnitude of correlation, for a given frequency (or band), between different areas of the 
cerebral cortex. Thus, high coherence and/ or PLV implies substantial communication 
between different cortical areas while low coherence and/ or PLV reflects regional 
autonomy or independence (Nunez & Srinivasan, 2006). 

3 Non-Invasive Functional Brain Biomarkers of Human Sensorimotor 
Performance: 

Although the signal processing approaches described above are applicable to both EEG and 
MEG signals, we will focus mainly on brain biomarkers derived from EEG since, as 
mentioned in the introduction, this technique is portable and therefore is particularly well 
suited for ecological motor tasks such as aiming (e.g., marksmanship, archery), drawing, 
arm reaching and grasping task. Therefore, most of the examples below will present the 
results of brain biomarkers derived from EEG signals. 
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3.1 Spectral power 

A series of studies that began in the early 80' s provided a growing body of evidence that it is 
possible to assess the cortical dynamics of motor skills in novice and expert performers 
during visuomotor challenge such as marksmanship and archery tasks. These studies 
revealed changes in EEG activity with skill learning as well as differences in EEG power 
between novice and expert sport performers (Del Percio et al., 2008; Hatfield et aL, 1984, 
2004; Haufler et al, 2000; Kerick et al, 2004; Landers et al, 1994; Slobounov et al, 2007). 
Specifically, the power computed for the alpha and theta frequency bands were positively 
related to the level of motor performance (Del Percio et al., 2008; Hatfield et al., 2004; 
Haufler et al, 2000; Kerick et al, 2004). 
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Fig. 4. Mean EEG power (mV 2 ) spectra (1-44 Hz) at left and right homologous sites in the 
frontal and temporal regions during the aiming period of the shooting task for expert 
marksmen versus novice shooters (Adapted from Haufler et al., (2000) with permission from 
Elsevier Science). 



For instance, Haufler et al., (2000) showed that, compared to novices, experts revealed an 
overall increase in EEG alpha power in the left temporal lobe (i.e., T3) while the same 
comparison between novices and experts performing cognitive tasks that were equally 
familiar to them did not provide any differences. The authors concluded, therefore, that the 
EEG alpha power differences observed were likely due to the difference of level in mastery 
of the motor task (see Fig. 4). Obviously, the differences in cortical dynamic between novices 
and experts revealed by these studies were accompanied with important differences 
between performances (i.e., the novices scored lower and exhibited more variability in their 
performance than the experts). Thus, these studies provided brain biomarkers (e.g., alpha 
power) able to identify a high level of motor performance resulting from an extensive 
practice period, without, however, considering the changes of such brain biomarker 
throughout the training period itself. 

Interestingly, in a more recent study Kerick et al., (2004) extended these investigations by 
assessing the dynamic changes throughout a marksmanship intensive training for novices 
during three months. The results revealed that, throughout the training, the performance for 
the shooting task was enhanced (Fig. 5 A) concomitantly with an increased EEG alpha power 
(Fig. 5B) at the temporal level located on the contralateral side (i.e., T3, left temporal lobe) 
while such observation was not observed when the subjects were at rest. Such EEG changes 
are generally interpreted as indicative of high levels of skill and associated with a cortical 
refinement leading to reductions of nonessential cortical resources (Hatfield & Hillman, 
2001). This kind of neural adaptation process may result in simplification of neurocognitive 
activity and less possibility of interference with essential visuomotor processes. Within an 
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activation context, a decrease in alpha power frequency band (i.e., desynchronization) 
represents an activated cortical site. Conversely, an increase in alpha power (i.e., 
synchronization) corresponds to a reduction of activation of a given cortical region 
(Pfurtscheller et al., 1996) indicating a decrease of the recruitment of neural resources. 
In addition to the alpha frequency band, several studies suggested that theta oscillations are 
also related to performance enhancement (Caplan et al., 2003; Tombini et al., 2009). For 
instance, during a virtual maze navigation task, Caplan et al., (2003) observed that theta 
oscillations reflected an updating of motor plans in response to incoming sensory 
information that facilitates the information encoding of participant's cognitive map. 
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Fig. 5. A. Shooting percentages by practice session. The slope of the linear regression 
revealed a significant increase in performance over all practice sessions from time 1 to 3 
(equation lower right corner). The different symbols represent the performance scores of 
individual participants on separate days of practice. B. Changes in mean power from time 1 
to 3 during shooting (SH), postural (PS), and Baseline (BL) condition (T3, left panel; T4, right 
panel). (Adapted from Kerick et al., (2004) with permission from Wolters 
Kluwer/Lippincott Williams). 

Although other interpretations of theta power increases are plausible (e.g., frontal theta EEG 
synchronization could also reflect an increased short term memory load; for a review see 
Klimesch et al., 2008), a growing body of work suggest that theta oscillations are 
functionally associated with error monitoring (Cavanagh et al., 2009; Larson & Lynch, 1989; 
Smith et al, 1999; Yordanova et al, 2004). 

Thus, taken together these studies suggested that changes in alpha and theta power can be 
used as non-invasive functional brain biomarkers capable either to assess the level of 
mastery of a given sensori-motor task (e.g., marksmanship task) and/ or to track the brain 
status during motor practice. However, these studies used visuomotor task where upper 
limb movements were extremely specific (e.g., archery, marksmanship task) without 
considering more familiar movements used in daily activities such as arm reaching, 
grasping and tool or object manipulations. Moreover, these investigations addressed the 
improvement of an established motor ability (e.g., Haulfer et al., 2000), or a long learning 
period of a skill involving no interference with previous motor experience (e.g., Caplan et 
al, 2003; Kerick et al, 2004). Interestingly, Kranczioch et al, (2008) showed that the learning 
of a visuomotor power grip tool led to EEG changes in spectral power and cortico-cortical 
coupling (i.e., coherence). However, this study did not involve a tool that required 
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suppression of a familiar response. Nevertheless, in daily activities, we frequently need to 
adapt our motor commands related to our upper limb to learn new input-output mappings 
characterizing novel tools by inhibiting familiar behavior or responses that are no longer 
valid to manipulate them. Such tool learning requires the selection and guidance of 
movements based on visual and proprioceptive inputs while frontal executive function 
would inhibit the pre-potent input-output relationships during acquisition of the internal 
model (also called internal representation) of the new tool. This would be typically the case 
if a person has to learn to manipulate a new tool such as a neuroprosthetic. It should be 
noted that Anguera et al., (2009) used a visuomotor adaptation task requiring suppression of 
preexisting motor responses in order to quantify the changes in error-related negativity 
associated with the magnitude of the error. However, this study did not focus on tracking 
the learning process by using brain biomarkers derived from spectral power and/ or phase 
synchronization. 

Based on this rational, a recent study (Gentili et al., 2008) intended to address this problem 
by analyzing the cortical dynamics during the learning of a new tool having unknown 
kinematics features. In this experiment, fifteen right-handed healthy adults subjects sat at a 
table facing a computer screen and, with their right hand, had to perform "centre-out" 
drawing movements (on a digitizing tablet) linking a central target and one of four 
peripheral targets. Movement paths were displayed on the screen, but a horizontal board 
prevented any vision of the moving limb on the tablet. EEG signals were acquired using an 
electro-cap with 64 tin electrodes, which was fitted to the participant's head in accordance 
with the standards of the extended International 10-20 system (Fig.6). First, the subjects 
performed 20 practice trials at the beginning of the experiment in order to be familiarized 
with the experimental setup. After this familiarization period, the experiment was divided 
into three sessions: i) pre-exposure, ii) exposure and iii) post-exposure. During the pre- and 
post-exposure phases the subjects performed, under normal visual conditions, 20 trials (i.e., 
1 block). During the exposure phase, (180 trials, i.e., 20 trials x 9 blocks) ten subjects (i.e., 
learning croup) had to adapt to a 60° counter clock-wise screen cursor rotation. In addition, 
five healthy (i.e., control group) subjects were examined using the same protocol but in the 
absence of any visual distortion. Movements were self-initiated and targets were self- 
selected one at a time. All the targets were displayed throughout each trial. The instructions 
were to draw a line as straight and as fast as possible linking the home target and the 
peripheral target. Unknown to the participants, a trial was aborted and restarted if the time 
between entering the home target and movement onset was less than 2s. Therefore, 
participants had enough time to both select the target and plan their movement providing, 
thus, an extended time-window to analyze cortical activations related to preparation 
processes (i.e., planning) of the movement. 

In order to quantify the motor performance during both movement planning and movement 
execution periods, the Movement Time (MT), Movement Length (ML) and Root Mean 
Square of the Error (RMSE) were computed from the 2D horizontal displacements. The MT 
was defined as the elapsed time between leaving the home circle and entering the target. 
The ML was defined as the distance traveled in each trial. 
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Fig. 6. Experimental device to record kinematics and EEG signals during the visuomotor 
adaptation task. Subjects sat at a table facing a computer screen located in front of them at a 
distance of ~60 cm and had to execute the motor task which consisted of drawing a line on a 
digitizing tablet (represented in light blue on the figure) that was displayed in real-time on 
the computer screen. The home target circle was the origin of a direct polar frame of 
reference, and the target circles were positioned 10 cm from the origin disposed at 45°, 135°, 
225°, and 315°. Once a successful trial was performed, to prevent any feedback, all visual 
stimuli were erased from the screen in preparation for the next trial. 

The RMSE was computed to assess the average deviation between the movement trajectory 
from the 'ideal' straight line connecting the home and the pointing target. For the nine 
learning blocks, the mean and standard deviation of the ML and MT were computed. In 
order to take into account any differences in subject's performance during the pre-exposure 
phase (i.e., baseline condition) and to focus on changes due solely to adaptation, the MT, ML 
and RMSE values were standardized with respect to the pre-exposure stage. 
Continuous EEG data were epoched in 2-s windows centered at movement onset. Both pre- 
(i.e., planning) and post- (i.e., execution) movement time-windows were considered. Single- 
trial data were detrended to remove DC amplifier drift, low-pass filtered to suppress line 
noise, and baseline-corrected by averaging the mean potential from -1 to 1 s. The EEG 
signals were cleaned by means of the ICA Infomax method applied on a single-trial basis 
described in section 2.1.1. For each subject and each single-trial, the EEG power (ERS/ERD) 
were computed by squaring and integrating the output of a dual band-pass Butterworth 
fourth order filter, and standardized with respect to the pre-exposure stage. The EEG power 
was computed for the alpha (low: 8-10 Hz, high: 11-13 Hz), beta (low: 13-20 Hz, high: 21-35 
Hz); theta (Low: 4-5 Hz, High: 6-7 Hz) and y (36-44 Hz) bands. The entire alpha, beta and 
theta frequency bands were also analyzed. For the alpha band, two similar frequency ranges 
have been considered, i) alphal: spread form 8 to 13Hz, and ii) alpha2: spreads from 9 to 13 
Hz. For each sensor and each block, the average power changes (across subjects) were fitted 
using a linear model from which the coefficient of determination (R 2 ) and its slope were 
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obtained. The sensors that showed a fit indicating a coefficient of determination capable to 
explain at least 50% of the variability of the data (i.e., R 2 >0.50) allowed us to determine the 
sensor clusters and the frequency bands of interest. The results of this procedure led us to 
consider the two alpha frequency bands and the high component of the theta frequency 
band for the right (FT8, T8, TP8) and left (FT7, T7, TP7) temporal and right (FP2, AF4, F4, F6, 
F8) and left (FP1, AF3, F3, F5, F7,) frontal lobes. This procedure led us also to consider the 
two alpha frequency bands for the left (PI, P3, P5, P7, P03, P05, P07) and right (P2, P4, P6, 
P8, P04, P06, P08) parietal and left (Ol) and right (02) occipital regions (For the electrodes 
sites see Fig. 6). It must be noted that the results for both alpha bands were similar. 
However, since the findings for the second alpha band (i.e., [9-13Hz]) were slightly better 
only this frequency band will be presented and discussed. For the alpha (i.e., [9-13Hz]) and 
high theta (i.e., [6-7Hz]) bands and the eight clusters of interest, the average power values 
were computed, and the same fitting process was applied. Furthermore, in order to 
investigate any correlation between the kinematics data and the EEG power, the average 
EEG power values obtained for the clusters of interest were plotted versus the MT, ML and 
RMSE values. Exponential (single and double), linear and quadratic models were used to fit 
these relationships. The best fit was selected by considering the coefficient of determination 
and its adjusted value, the mean square error of the fit, and the sum of squares due to the 
fitting error. 

The results showed that, during the early learning phase, the subjects performed distorted 
movement trajectories with a slow progression towards the targets. However, as the subjects 
of the learning group learned the unknown physical (kinematics) properties of the novel 
tool, the analysis of the motor performance revealed that the MT, ML and RMSE decreased 
throughout adaptation (Fig. 7A-C). From the early to the late learning period, the trajectories 
were straighter and smoother while the control group did not show any performance 
improvement (Fig. 7A-C). 
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Fig. 7. Concomitant EEG and kinematic changes throughout learning for the learning and 
control groups. (A) Changes in MT (±SE) throughout the learning blocks. (B) Changes in ML 
(±SE) (purple) and RMSE (±SE) (blue) throughout the learning blocks. (C) Changes in 
average trajectory (thick black lines) throughout learning for early, middle and late exposure 
(the grey area represents the standard error across subjects). (D) Qualitative EEG changes in 
alpha (first and third row) and high theta (second and fourth row) frequency bands for the 
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frontal, temporal, parietal and occipital regions during planning (two first rows) and 
execution (two last rows). For the sake of clarity, sensors which did not belong to the 
clusters of interest were set to the minimal value of the scale for the scalp plot. The results of 
the learning group and control group are represented in the left and right column, 
respectively. (Adapted from Gentili et al., (2008) with permission from EURASIP). 

Simultaneously to these behavioral changes, the results revealed that, as the subject adapt, 
the alpha and the high component of the theta power increased in the frontal and temporal 
lobes whereas an increased in alpha power also took place in the parietal lobes. Moreover, 
these spectral changes occurred during both movement planning (i.e., movement 
preparation) and movement execution. It must be noted that this alpha frequency band 
spread form 9 to 13Hz showed the largest reactivity during the adaptation to the novel tool 
and thus provides a better brain biomarker. Contrary to the learning group, the control 
group did not exhibit any changes in spectral power (Fig. 7D). 
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Fig. 8. Linear fits of EEG power changes for the frontal and temporal clusters for the 
participants of the learning group. Standardized values of the average EEG power 
computed across subjects (n=10) of the learning group and blocks (n=9) for the alpha and 
the high theta frequency bands recorded from the right (FT8, T8, TP8) and left (FT7, T7, TP7) 
temporal lobes and right (FP2, AF4, F4, F6, F8) and left (FP1, AF3, F3, F5, F7) frontal lobes. 
The blue and red stars indicate that the slopes were significantly different from zero for 
planning and execution, respectively. The black star indicates that the slopes between 
planning and execution were significantly different. The two bars on the right side of each 
panel represent the average value of the EEG power for the same cortical sites and the same 
frequency band for planning (blue) and execution (red) of the control group. (Adapted from 
Gentili et al, (2008) with permission from EURASIP). 
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Among the various models tested to fit these spectral changes, the best model that was able 
to capture these changes was linear. Only the left temporal lobe presented a significantly 
linear increase for the high component of theta power during movement planning (Fig. 8A). 
However, for the frontal lobes, the same linear theta power increase occurred during both 
movement planning and execution with similar slopes (Fig. 8C). For both the temporal and 
frontal lobes, the alpha power significantly increased linearly during both movement 
planning and execution. The slopes were also different between movement planning and 
execution (Fig. 8B, D). Finally, the alpha power showed a significant linear increase in the 
left and right parietal lobes for the planning while only a tendency was observed for the 
execution and both movement stages for the two occipital lobes (Fig. 9 A, C). 
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Fig. 9. Linear fits of EEG power changes for the occipital (A) and parietal (B) clusters for the 
learning group. Standardized values of the average EEG power computed across subjects 
(n=10) and blocks (n=9) for the alpha frequency bands recorded from the right (02) and left 
(Ol) occipital lobes and right (P2, P4, P6, P8, P04, P06, P08) and left (PI, P3, P5, P7, P03, P05, 
P07) parietal lobes. The blue stars indicate that the slopes were significantly different from 
zero for planning. The two bars on the right side of each panel represent the average value of 
the EEG power for the same cortical sites and the same frequency band for planning (blue) and 
execution (red) for the control group. The scalp plot depicts the clusters of electrodes in the 
occipital and parietal sites (C) and also for the frontal and temporal sites (D). For both panels, 
the blue and red circles indicate that the linear models for the alpha and theta power showed a 
coefficient of determination (R 2 ) greater than 0.5 for the planning and execution of movement, 
respectively. The blue and red stars indicate that the linear models had a slope significantly 
different from zero for planning and execution phases, respectively. The black star indicates 
that the slopes for planning and execution are significantly different from each other. 



The previous results were obtained at a cluster level; however, a refined analysis conducted 
at the sensor level also showed that these linear changes where located on specific sensors 
(Fig. 9C, D) for these two frequency bands and both movement planning and execution. 
Finally, in order to find a correlation model between these spectral changes and those 
observed in kinematics during performance several models have been tested. 
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Fig. 10. Changes in EEG power in the alpha and high theta bands versus kinematics. The 
first two rows represent the average values of the standardized power of the alpha bands 
computed for the right and left temporal and frontal regions during planning and execution 
versus the concomitant changes in ML (first row) and RMSE (second row) for the learning 
group. The third row represents the same relationship for both alpha versus ML and high 
theta versus RMSE for the control group. (Adapted from Gentili et al., (2008) with 
permission from EURASIP). 



The findings showed that, among the models tested, the single exponential was able to 
capture with the best accuracy these co-variations between EEG power changes and the 
corresponding motor production (Fig. 10 A, B). The control group did not show any changes 
(Fig. 10C). 

Thus, it appears that these changes in theta and alpha power provide informative brain 
biomarkers to track the cortical dynamics in order to assess the level of performance and 
also to track during both planning and execution the level of mastery of a novel tool 
throughout learning. Although useful, this first type of brain biomarker has the drawback to 
be univariate, that is, the power computed at a particular scalp site is able to characterize 
activation patterns for a particular channel (or brain region) without accounting for 
functional network connectivity or communications between different regions of the cortex 
during performance. It must be noted that these spectral power changes have been robustly 
observed in EEG/ MEG studies and represent today a classical brain biomarker of human 
performance. Beside the spectral power, another type of brain biomarker, derived from 
EEG/ MEG, is the computation of the phase synchronization between two scalp sites. 
Although initially less popular, this second technique (see section 2.3) is increasingly used to 
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track the level of sensorimotor performance/ learning. Recently this approach led to 
interesting results that will be presented in the next section. 



3.2 Phase synchronisation 

Contrary to the previously mentioned investigations focusing on the spectral power analysis, 
there are only a few studies that analyzed the cortical networking by means of coherence 
and/ or PLV to assess the level of motor performance and/ or to track the learning dynamic. 
For instance, Bell and Fox (1996) reported a decreased EEG coherence in experienced infant 
crawlers relative to novice crawlers and attributed their findings to a pruning of synaptic 
connections as crawling became more routine. Another experiment, further directly related 
to our purpose and conducted by Deeny et al., (2003), compared EEG coherence between a 
frontal site (i.e., sensor Fz) and several other cortical regions in two groups of highly skilled 
marksmen who were similar in expertise, but who differed in competitive performance 
history. One of the two groups performed consistently better in competition and exhibited 
significantly lower coherence between the left temporal region (i.e., T3) and the premotor 
area (i.e., Fz) in the low-alpha (8-10 Hz) and low-beta (13-22 Hz) bandwidths during the 
aiming period (Fig. 11). 
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Fig. 11. Upper row. Expert and skilled group means for low-alpha (8-10 Hz) coherence 
estimates between Fz (premotor area) and frontal, central, temporal, parietal, and occipital 
sites in each cerebral hemisphere. Lower row. Expert and skilled group means for low-beta 
(13-22 Hz) coherence estimates between Fz (premotor area) and frontal, central, temporal, 
parietal, and occipital sites in each cerebral hemisphere. ""Significant difference, p <0.05; 
**T3-Fz coherence was significantly lower than T4-Fz coherence in the expert group only. 
(Adapted from Deeny et al., (2003) with permission from Human Kinetics Publishers). 



More recently, Deeny et al., (2009) confirmed that the coherence could also be useful to 
assess the brain dynamic in relation to the level of mastery of a motor task. Specifically, they 
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showed that experts generally exhibited lower coherence over the whole scalp compared 
with novices, with the effect most prominent in the right hemisphere. Coherence was 
positively related to aiming movement variability in experts (Fig. 12). 
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Fig. 12. A. Average variability of rifle aiming path during the 4 s prior to trigger pull in 1-s 
time bins for experts and novices. Error bars represent standard error. B. Coherence values 
for high alpha. C. Coherence values for low beta. ^Indicate significantly higher coherence in 
novice shooters relative to experts (p <0.05). C = central; F = frontal; O = occipital; P = 
parietal; T = temporal. (Adapted from Deeny et al., (2009) with permission from Heldref 
Publications). 

Taken together, the authors of these two studies suggested that these coherence results 
reflect a refinement of cortical networks in experts that was interpreted as a reduction of 
nonessential functional communications among the cortical regions of interest inducing in 
turn an improvement in motor performance. In other words, such coherence patterns 
provide brain biomarkers of specific motor planning as skill level increases allowing 
assessing the mastery level of a given task. As previously explained in the section related to 
the spectral power analysis, these studies assessed cortical dynamics for a well-established 
motor ability without addressing any learning manipulations of object or tool having 
unknown properties. As far as we know, only two investigations (Busk & Galbraith, 1975; 
Kranczioch et al., 2008) used coherence measurement to study learning during a visuomotor 
task. Specifically, Busk & Galbraith, (1975) reported decreased coherence between premotor 
(Fz) and motor (C3, C4) areas of the cortex and between the premotor and occipital regions, 
following practice on an eye-hand tracking task. More recently, Kranczioch et al., (2008) 
found changes in cortico-cortical coupling during learning of a visuomotor power grip tool. 
Specifically, they revealed that learning was variably associated with increased coherence 
between contralateral and/ or ipsilateral frontal and parietal, fronto-central, and occipital 
brain regions. However, the learning period was relatively short (e.g., only the early 
learning stage was considered in Busk & Galbraith, (1975)) and these studies did not involve 
the suppression of familiar behavior used in the daily life. 

By using the same tool learning protocol with unknown kinematics features (see section 3.1, 
Fig. 6), a recent analysis (Gentili et al., 2009b) aimed to identify any changes in phase 
synchronization between two electrode pairs using both spectral coherence and PLV. The 
aim was to extract information from these measures to provide additional non-invasive 
functional brain biomarkers able to track the sensorimotor performance while subjects 
learned to manipulate a novel tool. The pre-processing of the EEG, the choice of the 
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frequency bands of interest and the kinematics processing were similar to that previously 
described in section 3.1 for the same tool learning task. Both the spectral coherence and the 
PLV have been computed as mentioned in section 2.3. A visual inspection of the data led us 
to consider a linear and a logarithmic model to fit the relationship between the spectral 
coherence/ PLV changes and the kinematics parameters (MT, ML, RMSE) throughout 
learning. However, based on the criteria previously mentioned (see section 3.1), the 
logarithmic model allowed a better fitting of these relationships. It must be noted that, since 
for this experiment both spectral coherence and PLV provided similar results, thus, only the 
PLV results are presented in the following. The kinematics results are the same that those 
presented in section 3.1 (see Fig. 7A-C) indicating that the subjects learned to manipulate 
correctly the novel tool. 
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Fig. 13. Changes in PLV throughout the learning. A. Pair of electrodes showing a decrease of 
their synchronization throughout the learning during planning (top scalp plot) and 
execution (bottom scalp plot). B. Linear model capturing the changes in PLV during 
planning and execution for the pair of electrodes Fz-F3 (low alpha band), Fz-F4 (low beta 
band), Fz-C3 (low beta band) and Fz-Ol (gamma band). C. Linear model capturing the 
changes in PLV during execution for the pair of electrodes Fz-T7 (low theta band), Fz-P3 
(high alpha band), Fz-P4 (high alpha band), and Fz-F3 (high theta band). (Panels A and B 
reproduced from Gentili et al., (2009b) with permission from IEEE). 



While throughout learning the kinematics was enhanced (see Fig. 7A-C); 
electrophysiological changes in phase synchronization were simultaneously observed (Fig. 
13 A). Namely, as the subjects adapt, the electrodes pair Fz-F3 (low alpha band), Fz-F3 (low 
beta band), Fz-F4 (low beta band), Fz-C3 (low beta band) and Fz-Ol (gamma band) revealed 
a decrease captured by a linear model (i.e., R 2 >0.50) for both movement planning and 
execution (Fig. 13B). For planning, the slopes of these linear models were significantly 
different from zero (t-test, p<0.05) for Fz-F3 (low components of the alpha and beta bands), 
Fz-C3 (low beta band), Fz-Ol (gamma band) and during execution for Fz-F3 (low alpha 
band) and Fz-C3 (low beta band) while a trend was observed for Fz-F3 (low beta band, 
p=0.06) and Fz-F4 (low beta band, p=0.07). Also, for execution, the same analysis revealed 
that the electrode pairs Fz-T7 (low theta band), Fz-P3 (high alpha band), Fz-P4 (high alpha 
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band) and Fz-F3 (high theta band) showed a significant linear decrease of the PVL (t-test, 
p<0.05) throughout adaptation (Fig.l3C). 

Such linear decrease was correlated with an enhancement of the performance and 
particularly good logarithmic correlations were found between the changes in phase 
synchronization and the MT and ML parameters. The results for the correlation analyses 
showed that the relationships between the changes in PLV for the pairs Fz-F3, Fz-F4, Fz-C3, 
Fz-Ol and the MT and ML values were best fitted by using a logarithm (R 2 >0.40) for both 
planning and execution. The same correlation analysis performed for the pairs Fz-T7, Fz-P3, 
Fz-P4, Fz-F3 and the MT and ML values revealed that the same results were obtained 
(R 2 >0.50) only for movement execution. 
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Fig. 14. Representation of the PLV versus the MT (first row) and the ML (second row) for 
both movement planning (blue color) and execution (red color). A. Pair Fz-F3 (low alpha 
band); B. Pair Fz-C3 (low beta band); C. Pair Fz-Ol (gamma band); D. Pair Fz-T7 (low theta 
band); E. Pair Fz-F3 (low alpha band); F. Pair Fz-C3 (low beta band); G. Pair Fz-Ol (gamma 
band); H. Pair Fz-F3 (high theta band). Since the Pair Fz-T7 (low theta band) and Fz-F3 (high 
theta band) revealed a non significant linear decrease during planning, the fits for PLV 
values versus MT and ML are only presented for execution (see panel D and H). (Panels 
A,B,E,F reproduced from Gentili et al., (2009b) with permission from IEEE). 



As for the spectral power changes for the alpha and theta frequency bands, these changes in 
coherence/ PLV presented above, allow assessing the level of performance but also its 
development throughout a learning period. Therefore, the spectral power and 
coherence/ PLV provide brain biomarkers of the performance and learning in Human that 
may be useful in bioengineering/ biomedical applications, particularly for brain monitoring 
applications and/ or when the access to the actual performance is impossible. This will be 
presented in section 4, beforehand; the section 3.3 will present and discuss the advantages of 
these brain biomarkers but also their current limitations and the potential solutions to 
overcome them. 
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3.3 Strengths, weaknesses, and perspectives for brain biomarkers of the 
sensorimotor performance 

3.3.1 Strengths and weaknesses 

By revealing correlations between the spectral power, coherence/ PLV and motor 
performance, the research lines presented in this chapter provide potential non-invasive 
functional brain biomarkers to assess and track the level of performance and learning. It is 
important to note that these biomarkers are able to detect important differences in skills 
level such as those existing between novices and experts (e.g., Hatfield et al., 1984, 2004; 
Haufler et al., 2000) as well as to identify the learning dynamic related to different types of 
tasks inducing different neural resources (e.g., Gentili et al., 2008, 2009a,b; Kerick et al., 
2004). Moreover, although their scalp locations and frequency band of interest present slight 
variations from one task to another, it appears that these biomarkers share also some 
frequency (e.g., alpha band) and spatial (e.g., temporal region) features while being located 
on specific electrodes for the various tasks tested. Therefore, beyond certain specificities that 
are task-dependent, these biomarkers of human performance share a common consistent 
topology in term of frequency and spatial scalp locations across different tasks. Moreover, it 
must be noted that changes in phase synchronization for a specific frequency range do not 
necessarily imply similar power changes for the same electrodes (Kiroi & Aslanyan, 2006). 
Therefore, the availability of processing techniques for extracting and combining both 
univariate (i.e., spectral power) and multivariate (i.e., spectral coherence/ PLV) cortical 
measures might provide "multidimensional" brain biomarkers in the future. Such 
multidimensionality resulting from the combination previously described is expected to 
provide enhanced, robust biomarkers capable of tracking performance and learning 
dynamics, thus providing a potential solution to overcome limitations in current practical 
applications. This will be explained in the section 3.3.2. 

Another important point is directly linked to the fact that these biomarkers were derived 
from EEG during movement execution, but, more importantly, during movement 
preparation (i.e., planning; Deeny et al, 2003, 2009; Gentili et al, 2008, 2009a,b; Hatfield et 
al., 2004; Haufler et al., 2000). The availability of these biomarkers during movement 
execution and particularly during movement preparation (i.e., planning) involves two 
specific advantages. 

First, a biomarker of the performance during execution can be considered as a good 
complement of the behavioral measures available during and/ or after movement execution. 
More importanty, the presence of these brain biomarkers during planning also allow 
estimating/ predicting the on-coming performance level that is not available with usual 
peripheral and behavioral measurements. This important feature is common to many 
biomarkers such as the bispectral index derived from EEG used for the identification of 
anesthetic depth during pediatric cardiac surgery while the usual clinical signs are not 
accessible (Williams & Ramamoorthy, 2009). 

Second, the availability of brain biomarkers of the performance during movement 
preparation is a feature that becomes particularly important when considering overt but, 
more importantly, covert movement executions in the context of bioengineering and 
biomedical applications for rehabilitation. The expression "overt movement execution" 
corresponds to a movement actually performed such as those executed in daily activities. In 
this case, the person can see and feel his/her own limb moving. Conversely, the term 
"covert movement execution", also commonly named mental or motor imagery, refers to a 
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dynamic mental process during which a subject internally simulates a motor action without 
activating the muscles and, therefore, without any apparent motion of the limbs involved in 
that action (Gentili et aL, 2004, 2006; Jeannerod, 2001). Such motor imagery or covert 
execution is commonly used for mental practice/ rehearsal of specific performance skills, 
BCI approaches and more generally in rehabilitation (see section 4 of this chapter). 
Interestingly, many studies revealed that common neurocognitive mechanisms in terms of 
both similar neural structures and behaviour exist between overt and covert motor actions 
(Fadiga & Craighero, 2004; Gentili et aL, 2006; Jeannerod et aL, 2001). In particular, several 
investigations suggest that motor imagery involves the same neural mechanisms as those 
activated during preparation (i.e., planning) and execution of overt movements (e.g., 
Jeannerod, 1994, 2001). Therefore, although our task involved actual movements, since the 
present results suggest that these brain biomarkers are accessible during movement 
preparation, they may also be suitable for covert movement execution when a task is 
performed using motor imagery. 

Despite this research provided some interesting results and is still currently making 
progresses, two main limitations have to be considered. First, the present brain biomarkers 
of performance are based on a population analysis without considering subject individually. 
Second, their computation was based on the average value across several trials (e.g., 20 
trials). Definitely, considering the variability of the MEG/EEG signals from one trial to 
another and also the sensitivity of the EEG signal to environmental noise and artefacts, the 
approach consisting in defining brain biomarkers of the performance needs to investigate, to 
what extent these results can be extended when single subject and single trials are 
considered. This is important for future applications since they will be designed for single 
subjects and ideally based on single or eventually few trials. Recently, by using MEG 
applied to a similar tool learning task (described in Fig. 6), we started to address these two 
problems by analyzing the alpha power band ([9-13Hz]) in individual subjects using the 
same ERD/ERS techniques and testing different sliding window (e.g., length, overlap) 
across trials. The preliminary results suggest that, at the individual level, the spectral power 
for the alpha band ([9-13Hz]) computed at the frontal, temporal and parietal regions during 
movement preparation were able to predict the motor performance (Gentili et al. 2009a). 

3.3.2 Overcoming the current limitations by means of multiple constrains 

As suggested in section 3.3.1, a possible way to overcome the two main limitations 
previously mentioned (i.e., single subject and computation based on single or few trials) is 
to obtain robust multidimensional EEG/ MEG biomarkers able to assess the level of 
performance and learning by combining several individual biomarkers. In other words, the 
combination of several biomarkers would result in an increased number of conditions that 
have to be satisfied for estimating reliably any enhancement of the performance. The 
prediction problem is therefore constrained since a reliable estimation of performance needs 
to satisfy several constraints represented by the right combinations of biomarkers. For 
instance, if both a power increase and a coherence/ PLV decrease are simultaneously 
observed for specific frequency bands and brain regions, it seems reasonable to predict with 
a certain confidence that the subjects are successfully learning the task. Conversely, if we 
would have only one biomarker, this prediction would be less reliable. Therefore, the 
combination of several brain biomarkers such as phase synchronization and spectral power 
would provide cross-information resulting in the generation of robust and accurate non- 
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invasive brain biomarkers of the motor performance. This approach could also give insight 
into possible reasons for the failure of sensorimotor learning and adaptations. Thus, such 
multidimensional brain biomarkers might be better suited for applications based on 
individual subjects and single or few trials. 

It must be noted that, this first type of constraint was related to a combination of various 
biomarkers using the same brain imaging modality, i.e., EEG/MEG signals. However, 
another type of combination could also be considered by using the fusion across multiple 
recoding modalities in order to complement information provided from each imaging 
technique. For instance, in order to complement EEG/MEG signals analysis, fNIRS signals 
processing could provide additional brain biomarker by measuring the hemodynamic of 
brain activity. The choice to use fNIRS is guided by three reasons: First, although the 
hemodynamic activity has a lower temporal resolution than EEG, the fNIRS potentially 
provides more direct spatial resolution or localization abilities over EEG (Soraghan et al., 
2008). Thus, with the superior temporal resolution of EEG, merging these two techniques 
would allow for "the best of both worlds" (Coyle et al., 2007). Second, contrary to EEG, the 
hemodynamic response is influenced by head/ body orientation with respect to the 
gravitational axis whereas fNRIS signal is relatively less sensitive to artefact and 
environmental noise than EEG. Once again, since both do not have these two common 
weaknesses their combination appears to be advantageous. Third, although fNIRS only 
penetrate the cortex relatively superficially (-2.0 cm; Rolfe, 2003) contrary to classical fMRI, 
these signals can be recorded by portable devices as it is also the case for EEG, making them, 
particularly well suited for applications in practical/ ecological situations with various 
populations (e.g., healthy persons, patients, children, elderly, military personnel, etc.). It 
must be noted that the idea to combine several biomarkers within (power, coherence/ PL V) 
and between (fNIRS) imaging modalities has already been proposed for clinical applications 
(Guarracino et al., 2008) such as for brain injury prediction (Ramaswamy et al., 2009) and 
amyotrophic lateral sclerosis (Turner et al., 2009). From a practical point of view, this signal 
fusion across multiple imaging modalities could ideally be performed by using a recoding 
system that embed both EEG and fNIRS sensors. 

3.3.3 Emotional states on brain biomarkers of the performance 

A question that is naturally raised is the influence that some psychological and mental states 
such as emotion, stress or fatigue could exert over the quality of sensorimotor performance. 
If such adverse psychological and mental states disrupt the motor performance, it is 
legitimate to wonder to which extent the biomarkers tracking this same performance would 
also be affected. However, the majority of the performance stress-related studies focus on 
behavioural aspects without analyzing the cortical dynamics (Staal et al., 2004). Ongoing 
research by Hatfield and colleagues is beginning to provide some insight into such 
questions by placing performers under stressful conditions. For instance, Rietschel et al., 
(2008) asked participants to perform a marksmanship task under both regular performance- 
alone and competitive conditions. Changes in the Spielberger State Anxiety Inventory 
(STAI), heart rate, Cortisol and skin conductance evidenced an increased state anxiety during 
the competitive condition. Furthermore, the performance was affected during the 
competition along with a significant decrease in alpha power. Similarly, when subjects 
performed a drawing movement task under high level arousal conditions they exhibited 
higher levels of coherence associated with decreases in performance (Rietschel et al., 2006). 
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Therefore, these results provide evidence that the brain biomarkers of sensorimotor 
performance can be disrupted by psychological and mental states such as emotion, stress. 
Thus, from a physiological point of view, it is possible to consider that an increased degree 
of stress would induce the recruitment of nonessential neural resources during task 
execution, leading to a reduction of cortical refinement (i.e., a reduction of alpha power and 
an increase in cortico-cortical communication) that reflects sub-optimal performance. In 
other words, we could consider that, to some degree, the brain biomarkers are contaminated 
with a sort of noise. However, even in this case, they may still be informative since in some 
instances they could also unravel the possible causes (e.g., stress, fatigue) of alterations in 
behavioral performance which cannot be revealed by peripheral motion parameters (e.g., 
kinematics) alone. For instance, in the study where subjects learn a novel tool, the absence of 
learning/ adaptation could also be due to fatigue. Nevertheless, when considering the 
spectral power, the frontal biomarkers evidenced here are neither in the same spatial 
location (frontal midline) nor in the same frequency band (low theta band) than the fatigue- 
related EEG power (Makeig et al., 2000; Oken et al., 2006). Similarly, when considering the 
coherence/ PL V, factors such as stress or fatigue imply an increase and not a decrease in 
phase synchronization and is generally identified for different electrodes pairs and/ or 
frequency bands (Andersen et al., 2009; Lorist et al., 2009) than those found in the tool 
learning study (see section 3.2). Therefore, this clearly illustrates: i) the advantage to 
combine different biomarkers of the performance to obtain more robust predictions, ii) the 
benefit to combine them with other biomarkers identifying some adverse mental states (e.g., 
fatigue, stress) to be able to better decipher or indicate potential causes of a poor learning 
performance. Futures research should provide insights about these various possibilities, 
their benefit and limits. 



3.3.4 Fusion of structural and functional brain biomarkers 

Although the two previous sections (3.3.2 and 3.3.3) focused on different problems, both of 
them emphasized the importance for cross-information by combining several biomarkers. 
Indeed, it can be reasonably expected that such combination of biomarkers would lead to a 
robust tracking of motor performance and learning. It must be noted that such a 
combination can be performed not only between functional biomarkers but also between 
both structural and functional biomarkers. For instance, biomarkers can predict the 
performance based on information at the genetic/ molecular level (e.g., naloxone, Cortisol) or 
from behaviour such as heart rate or skin conductance (Armstrong & Hatfield, 2006). Thus, 
such convergence between these biomarkers, different in nature, would allow performing 
an even more robust prediction to assess accurately the level of performance and to 
track/ predict precisely the learning dynamic. Although this chapter introduced mainly the 
concept of functional brain biomarkers for performance assessment, it appears clearly that 
both structural and functional brain biomarkers must be seen as a complementary source of 
information. Interestingly, while structural brain biomarkers using methods form genetic 
may be more appropriate on a long timescale prediction such as very early diagnostic, 
functional biomarkers may be better suited for short timescale prediction such as a real-time 
tracking of the neural events. Such combination of structural and functional brain 
biomarkers is an emerging research line. For instance, recently Deeny et al., (2008) 
investigated MEG measurements in relation to genetic markers such as the epsilon4 allele of 
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the apolipoprotein, providing a method to detect risk factors for Alzheimer's disease 
(Corder et al, 1993). 

4. Current Brain Biomarkers for Sensorimotor Performance and 
Bioengineering Applications 

Beyond the considerations presented in section 3, the techniques presented to record and 
process brain biomarkers non-invasively using portable systems make them particularly 
well suited for real-time (or close to real-time) prediction in practical/ ecological 
applications. Although multiple potential applications can be considered for the future, this 
section will illustrate two possible applications. The first one will be the design of future 
smart neuroprosthetics by proposing solutions to overcome some well-known BCI-related 
problems. The second application (that is actually to some extent a generalization of the first 
one) will be related to brain monitoring in the context of overt and covert movement 
execution to accelerate learning or re-learning when a task is performed/ learned using 
actual movements and/ or motor imagery. 

4.1 Neuroprosthetic applications: towards a smart Brain Computer Interface 

The changes previously described in EEG power and coherence/ PLV that mirror human 
motor performance may potentially provide powerful biomarkers for tracking human 
learning/ adaptation status when one has to learn/ adapt to a new tool. A first potential 
interesting role of these brain biomarkers would be to overcome the well-known difficulties 
related to BCI systems such as adaptive decoding, constant recalibration and the 
maintenance of stable performance while a user tries to control a neuroprosthesis (Vaughan 
et al., 2003). Traditionally, motor-imagery-based BCI approaches are divided into two 
phases. The first one consists of a calibration phase to determine the parameters of a 
decoding algorithm, which has to map neural signals to a class of imagined movement. The 
second phase aims to train the subject by providing him/her sufficient feedback to change 
his/her cortical dynamics in order to control an external device via the BCI system. It is 
important to note that during this second stage, since the adaptation depends on the 
capacity of the user's brain to change its cortical dynamics, frequent recalibrations of the 
decoding algorithms are required when the user's performance degrades (Blankertz et al., 
2009). In order to address these problems, some solutions have been proposed and notably 
by means of adaptive algorithms (Blankertz et al., 2006; Sykacek et al., 2004). However, these 
approaches use supervised adaptation based on a priori knowledge of an external target. 
Although helpful, the requirement of such a priori information actually represents a major 
pitfall for practical BCI applications since the user should decide when and where to direct 
his/her intentions. In other words, no information of external targets is available to the 
decoding algorithm (Blankertz et al., 2006; Vidaurre et al., 2007). The complexity of using 
two adaptive controllers (the user's brain and the decoding algorithm) is not new and has 
been already raised (McFarland et al., 2006; Vaughan et al., 1996); however, it continues to 
be an issue, and no satisfying solutions of this problem have been provided (McFarland et 
al., 2006). The brain biomarkers of performance presented in this chapter may help to 
overcome such important drawbacks of BCI. Indeed, such biomarkers could be used to 
continuously adapt the decoding algorithm to the subject's mental states, thereby allowing a 
stable co-adaptation/ cooperation between the user and the BCI system. This is especially 
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relevant when the user has to learn the physical properties of a new tool and/ or a novel 
environment as is the case when a user intends to control a neuroprosthetic device. For 
example, the alpha power at the frontal, temporal and parietal sites combined with 
coherence/ PLV for the low beta frequency bands between the pair of electrodes Fz-F3 and 
Fz-C3 could be computed using a sliding window (e.g., 15-20 trials). If the user's brain 
considerably adapts as indicated by an increased alpha power combined with a reduced 
coherence/ PLV at the brain sites mentioned above, then the BCI decoding algorithm should 
not update its parameters. Conversely, it should adjust the parameters, by using, for 
instance, a reinforcement learning signal, to compensate for a user's poor performance (in 
that case reflected by a decreased alpha power and an increased coherence/ PLV at the brain 
sites mentioned above). 

As previously mentioned in section 3.3.3, the use of such biomarkers could also reveal the 
sources of alteration in behavioral performance which cannot be revealed by kinematics 
parameters alone. For instance, poor learning/ adaptation performance could be due to 
other factors such as stress or fatigue. These biomarkers, thanks to their specificities in term 
of scalp sites and frequency bands (and also with eventual additional information such as 
hemodynamic response provided by fNIRS), could reasonably unravel the possible origin of 
poor motor learning, providing, therefore, relevant covert supervision of the user during 
BCI training. For example, in practical use, it is important to decipher if a user's poor BCI 
performance is related to fatigue or to bottlenecks related to information processing guiding 
the algorithm to adapt to the user's cognitive state, which is usually impossible to access 
from behavior. 



4.2 Brain monitoring applications 

Another possible application of functional brain biomarkers would be related to brain 
monitoring for overt and more importantly for covert execution. It is well known that motor 
imagery, or covert execution, share a lot of functional commonalities and that many neural 
structures are commonly activated during both overt and covert movement. On the other 
hand, there is also a growing body of evidence that suggests that it is possible to learn, or at 
least improve, performance with practice using motor imagery also called mental training. 
Most of the studies focusing on mental practice either considered performance enhancement 
in a healthy population (e.g., Gentili et al., 2006; Yaguez et al., 1998) or a rehabilitation (e.g., 
Jackson et al., 2004; Page et al., 2001) context where a positive effect on subsequent actual 
motor performance was evidenced. While it is possible to assess the effects of such covert 
practice on subsequent actual movements, it is impossible to continuously monitor mental 
training (unless a trial is actually executed) since no overt execution is available. However, 
the brain biomarkers presented here would allow for assessing the level of performance 
during mental training and tracking of learning dynamics. Such brain biomarkers could be 
coupled to a neurofeedback system providing, thus, an enhanced feedback of performance 
during overt execution (in addition to classical feedback) or covert execution where usually 
no feedback is available. Such brain monitoring systems for covert/ overt movement 
execution would allow efficient supervision of performance, resulting in an accelerated 
learning or re-learning. Such bioengineering systems could be applied in various 
populations ranging from military personnel desiring to rapidly acquire skills to any 
persons subjected to a motor impairment undergoing rehabilitation where enhanced 
guidance for both patient and therapist would be beneficial. It must be noted that these 
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biomarkers would allow monitoring and fitting of the training time-scale for each individual 
since it is reasonable to expect that two individuals will not mentally learn at the same 
speed. For instance, for the same task some individuals using mental practice may need 40 
trials to reach acceptable performance while others would need 60 trials to reach the same 
level of performance. However, it is not possible to detect any progression in performance 
when using motor imagery (except by occasionally using actual execution) unless we use 
these brain biomarkers to create a customized training timescale for each individual. 
Moreover, as for BCI application, it would also be possible to know if a poor performance is 
related to sensorimotor learning processes or induced by some adverse mental states such as 
fatigue. Thus, the therapist could adapt the current rehabilitation session to the patient's 
cognitive state in order to improve training efficiency without having to access behavioral 
measures. 

At present, the current research focuses mainly on brain biomarkers for healthy people since 
a well-established model of these brain biomarkers needs to be defined before moving 
towards practical applications for pathology in a rehabilitation context. It is of interest to 
consider if such brain biomarkers would be applicable for patients subjected to neural 
pathologies. Although these biomarkers should be affected by a given pathological state, it 
is still possible to find their modified version adapted to this pathology as a BCI decoding 
algorithm is able to map a pathological neural activity to the desired output (Neuper et al., 
2003). This would necessitate applying the same techniques and approaches, albeit with 
some modifications, to provide biomarkers engineered for specific neural pathologies. For 
instance, it has been suggested that mental imagery practice would have positive effects on 
persons subjected to cerebral palsy (Trusceli et al., 2008; Zabalia, 2002). Therefore, under 
such conditions, the cerebral palsy-specific performance biomarkers would allow 
monitoring of the brain to provide feedback for a therapist in order to accelerate and 
improve performance and, thus, the physical therapy process. It must be noted that, beyond 
application, such brain biomarkers could also provide useful information about the cortical 
neural networks of patients suffering from neural diseases. Still taking the example of 
patients with cerebral palsy, specifically, these brain biomarkers could provide insights into 
the effects of physical therapy by, for instance, estimating the benefit of motor imagery on 
reorganization of cortical dynamics and the degree of automatization of the movement. 
Namely, the coherence/ PLV biomarker (Busk & Galbraith, 1975; Deeny et al, 2003, 2009; 
Gentili et al., 2009b) may be of particular interest to analyze any possible changes in cortical 
network recruitments throughout the rehabilitation procedure associated with any potential 
motor performance improvement. Moreover, several investigations have suggested that an 
increase in alpha power in the temporal, frontal regions would reflect that movement 
become more automatized as a function of practice, requiring less attentional and processing 
resources, since as strategies and skills are developed, there is a less extensive cortical 
contribution to task performance, resulting in increased alpha power (Gentili et al., 2008, 
2009a; Hatfield et al., 2004; Smith et al., 1999). Therefore, when using mental imagery the 
computation of such spectral power could provide a biomarker able to assess the degree of 
automatization of the repeated actions throughout a rehabilitation session. Finally, as 
previously mentioned, a multidimensional brain biomarker could be even more effective by 
combining information such as the spectral power, coherence/ PLV and hemodynamic 
responses using fNIRS. 
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5. Conclusions and Perspectives 

Nowadays, some non-invasive functional brain biomarkers able to assess cognitive- 
motor/ sensorimotor performance and learning level are available. However, they were 
mainly analyzed by means of investigations based on populations of subjects. The next 
challenge is to generalize these biomarkers to single subjects using single or few trials in 
tasks using actual movements or motor imagery. In order to reach these new aims, further 
research is needed to provide multidimensional biomarkers by considering the fusion of 
both processing techniques (e.g., EEG/MEG spectral power and coherence) and the nature 
of neural signals (e.g., hemodynamic response with fNIRS). Such approaches are expected to 
provide robust models for these biomarkers. Today, these brain biomarkers are engineered 
based on healthy people; however, in the future these methods could be transferred to 
alleviate neural disorders, provide new types of smart neural prostheses, and create brain 
monitoring tools to allow the emergence of a new generation of assistive technology for both 
healthy (e.g., accelerated learning) and pathological (e.g., rehabilitation) human populations. 
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1. Introduction 



Audible sound analysis is assumed to be an integral part of any speech processing system, since 
the audible frequency ranges are naturally used for vocal communications. Inaudible sound, 
either ultrasonic or subsonic, is less widely researched - although sub-sonics (also called 
infrasound) have found a niche application in sound strengthening for spatial effects (Begault, 
1994) and more immersive audio experiences (Kyriakakis, 1998) since there is evidence that these 
lower frequencies can be felt even if not heard. Higher frequency ultrasound is commonly used 
for diagnostic imaging in both medical and engineering fields (Szabo, 2004), and even sometimes 
for medical treatment (Haar, 1999). Due to the required resolution of these applications, they tend 
to operate at frequencies above the MHz range. In the animal kingdom, the majority of 
communication signals share a similar frequency range with humankind, although extending 
downward to infrasound in sea creatures such as whales (Clark, 2004), and upward to 
ultrasound in bats, moths, dolphins and so on (McLoughlin, 2009). 

One phonetically significant and growing application of ultrasound is ultrasonic speech, as a 
contribution of ultrasound to speech analysis and processing. The technology acts to 
augment the human natural speech production system, using data extracted from ultrasonic 
analysis to synthesize elements of audible speech. In this application, an ultrasonic signal in 
the kHz range is injected into the vocal tract (VT), which propagates and resonates through 
the vocal cavities and is emitted from the mouth as ultrasonic speech. This signal could 
potentially be converted to audible speech by appropriate down-conversion. 
This chapter introduces ultrasonics, particularly low frequency (LF) ultrasonic waves, and 
analyses their interaction with the human vocal tract. Ultrasonic frequencies used in this 
application are relatively low (near-audible), extending upwards from the upper threshold 
of human hearing to around 100 kHz in frequency. These frequencies are easily generated, 
captured and recorded by much existing audio equipment - and can be readily processed in 
real-time with available hardware. 

In this chapter, the basic principles of ultrasonic speech technology and the most important 
issues concerning its implementation and signal processing are addressed. Most 
importantly, a modelling scheme describing ultrasonic behaviour within the VT is proposed 
and proven as the theoretical basis and the main framework of implementation of ultrasonic 
speech. This chapter concludes with open areas of research concerning the technology. 
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The scope of this chapter is as follows: In order to have a precise understanding of the 
problem, first the attributes of ultrasonic propagation are analyzed physically and 
mathematically in section 2. This section investigates these attributes, and describes linearity 
preconditions of any gas medium, the compliance with which, would allow ultrasonic 
propagation in that medium to be considered linear and lossless. 

Section 3 analyses the plausibility of the linearity assumption for the propagation of the low 
frequency portion of the ultrasound bandwidth in the VT by a numerical analysis of the 
impact of dispersion and attenuation of LF ultrasound and addresses issues such as exhaled 
C0 2 as a dispersive wave medium for ultrasound, losses and cross modes of resonance of 
the VT in such frequencies. 

Given this basic perspective, section 4 introduces ultrasonic speech as the usage of LF 
ultrasound for speech processing, surveys previous implementations of the technology and 
describes the necessary requirements of the implementation. As in this method, the human 
VT is used to produce the ultrasonic output signal, there is a need to study the anatomy and 
physiology of human speech production system in general in section 5. The necessary pre- 
conditions for linear modelling in section 2 along with the numerical analysis of section 3, 
lead to the derivation of a linear source-filter model for the ultrasonic speech process in 
section 6. Many applications in the theory of speech processing rely on the classical source- 
filter model of speech production. Section 6 considers how this model can be adapted to 
ultrasonic wave propagation in the vocal tract by manipulating the sonic wave equations 
and deriving the vocal tract transfer function for ultrasonic propagation. 
At audible frequencies, linear predictive analysis (LP A) applies a linear source-filter model 
to speech production, to yield accurate estimates of speech parameters. Section 7 
investigates the possibility of extension of LPA to cover ultrasonic speech. Discussing some 
simplifying assumptions, the section leads to the application of LPA for the analysis of 
ultrasonic speech. By the extension of LPA to ultrasonic speech, we introduce the main set of 
features needed to be extracted from the ultrasonic output of the VT to be utilized in speech 
augmentation. The chapter then presents a concise outline of current research questions 
related to this topic in section 8. Section 9 finally concludes the discussion. 

2. Attributes of ultrasonic propagation 

Ultrasound can be defined as " Sound waves or vibrations with frequencies greater than 
those audible to the human ear, or greater than 20,000 Hz" (Simpson & Weiner, 1989). The 
starting point of the ultrasonic bandwidth resides implicitly somewhere between 16-20 kHz 
due to variations in the hearing thresholds of different people. The bandwidth continues up 
to higher levels 1 where it goes over to what is conventionally called the hypersonic regime 
(David & Cheeke, 2002). The upper limit of ultrasound bandwidth in a gas is around 1 GHz 
and in a solid is around 10 13 Hz (Ingard, 2008). At such mechanical vibrations exceeding the 
GHz range, electromagnetic waves may be emitted so that the upper limit of ultrasound 
may induce RF (radio frequency) electromagnetic waves (Lempriere, 2002). 
The general definition of sound indicates that "sound is a pressure-wave which transports 
mechanical energy in a material medium" (Webster, 1986). This definition can extend the 



1 which in a gas is of the order of the intermolecular collision frequency and in a solid is the 
upper vibration frequency (Ingard, 2008). 
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margins of understanding of sound beyond the hearing limitations of humans to cover any 
pressure wave including ultrasound. It has to be noted that similar to the sense of sight, 
which subjects the visible light region of the EM spectrum to special attention, the human 
sense of hearing has differentiated the "audio" segment of sound to be classically termed as 
"sound" in common language and other portions of the bandwidth have thus been 
classified in relation to the audible part as ultra or infrasound (similarly to visible light and 
infrared, ultraviolet terminology). 

The fact which should not be concealed is that the audible sub-band is only a tiny slice of the 
total available bandwidth of sound waves, and the full bandwidth, except at its extreme 
limits can be described by a complete and unique theory of sound wave propagation in 
acoustics (David & Cheeke, 2002). Accordingly all of the phenomena occurring in the 
ultrasonic range occur throughout the full acoustic spectrum and there is no propagation 
theory that works only for ultrasound. 

The theory of sound wave propagation in certain cases simplifies to the theory of linear 
acoustics which eases linear modelling of acoustic systems. It is generally preferential to 
approximate a system with a linear model where the assumptions of such modelling are 
plausible. Ultrasound inherits some of its behaviours from its nature of being a sound wave. 
There are also characteristics of the medium which impose some medium specific 
constraints on ultrasonic waves. Based on these facts we will review the general 
characteristics of ultrasound propagation as a sound wave and the effects of the medium, 
paying special attention to the required pre-conditions of linearity. 

2.1 Wave based attributes of sound 

Ultrasound as a sound wave, obeys the general principles of wave phenomena. The theory 
of wave propagation stems from a rich mathematical foundation of partial differential 
equations which are valid for all types of waves (Ikawa, 2000). In other words every wave, 
regardless of its production and physical detail of propagation can be described by a set of 
partial differential equations. All common behaviours observed in waves are 
mathematically proven by these equations (Rauch, 2008). 

To rest under the scope of generalization of the theory of waves, a physical phenomenon 
solely needs to fulfil the preconditions of being a wave by complying with the restrictions 
imposed by the wave equations. Afterwards the common behaviour of waves, proven 
mathematically for the solutions of these equations, would be valid for that specific physical 
phenomenon too. It has to be noted that although in today's understanding of waves we are 
quite confident that for example, sound "is" a wave, however compliance of each wave type 
with the wave equations as the necessary pre-condition, has long ago been proven by 
scientists of the corresponding discipline (Pujol, 2003). 

When the dimensions of the material are large in comparison to the wavelength, the wave 
equations become further simplified and can approximate the wave propagation as rays 2 . 
These simplified sets of wave equations are the basis of geometric wave theory (aka ray 
theory) of wave propagation (Buhler, 2006). The geometric wave theory permits freedom of 
microscopic details of wave propagation and describes the wave movement, reflection and 
refraction in terms of rays. The theory has been initially observed in optics and owes its 



2 A ray is a straight or curved line which follows the normal to the wave-front and 
represents the two or three dimensional path of the wave (Lempriere, 2002). 
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application to acoustic waves to (Karal & Keller, 1959; 1964) and has yielded geometric 
acoustics (Crocker, 1998) as the dual to wave acoustics (Watkinson, 1998). 
As a high frequency approximation solution to the wave equations, ray theory fails to 
describe the wave phenomenon in low frequencies when the wavelength is large compared 
to the dimensions of the medium. Consequently, in low frequencies we have to refer to 
general wave equations as the wave theory to describe the wave phenomenon. It has to be 
noted that wave theory is always valid but only in smaller wavelengths in comparison to the 
dimensions of the medium can the analysis be simplified by the geometric theory. 
In any case, because all the waves obey the same sets of partial differential equations, they 
have common attributes which are guaranteed by several principles extracted out of the 
wave equations. These principles manifest geometric and wave behaviour and are the 
general laws which impose similar conditions upon the propagation of waves in 
microscopic and macroscopic scales. The Doppler effect (Harris & Benenson et al., 2002), 
principle of superposition of waves in linear media (Avallone & Baumeister et al., 2006), 
Fermat's (Blitz, 1967) and Huygens principles (Harris & Benenson et al., 2002) are the 
fundamental laws of propagation for all the waves including ultrasound in wave and 
geometric theory. For interested readers, the mathematical derivation of some of these 
principles using wave equations is covered in (Rauch, 2008). 

For universal wave events such as diffraction, reflection and refraction which obey the 
general principles of wave propagation, there would be no exception to the general theory 
of sound propagation for ultrasound (David & Cheeke, 2002) except only the change of 
length scale which means that we have moved to different scales of the wavelength so the 
scale of material in interaction with waves and the technologies used for generation and 
reception of these waves will be different (David & Cheeke, 2002). 

2.2 Medium based attributes of sound 

The exclusive wavelength-dependant behaviours of ultrasound will present itself in the 
influence of the medium on wave propagation and we expect to observe some differences 
with audible sound where the wave propagation is apt to be influenced by characteristics of 
the medium through which it travels. In this section we consider the general attributes of a 
medium which impose special behaviours on a sound wave. Next in section 2.3 we will 
consider the effect of such attributes on ultrasound waves. When the medium of sound 
wave propagation is considered, the first important attribute under question is the linearity 
of the medium. Also important is a consideration of the attenuation mechanisms by which 
the energy of a sound wave is dissipated in the medium. 

2.2.1 Linearity 

Propagation of sound involves variations of components of stress (pressure) and strain in a 
medium. For an isolated segment of the medium we may consider the incoming wave stress 
as the input and the resulting medium strain as the response of the system to that input. To 
consider a medium of sound propagation as a linear system the stress-strain relation should 
be a linear function around the equilibrium state (Sadd, 2005). Gas mediums such as the air, 
match closely to the ideal gas law in their equilibrium state (Fahy, 2001) which states that: 

pv = nRT (1) 
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Where p is the gas pressure, v is the volume, t is temperature and n, R are constant 
coefficients depending on the gas. If one of the three variables of p, v or f remains constant, 
the relation of the other two, can easily be understood from (1) but sound wave propagation 
generally alters all of these three components in different regions of the gas medium. A 
general trend is to consider sound wave propagation in an ideal gas as an adiabatic process 
meaning no energy is transferred by heat between the medium and its surroundings when 
the wave propagates in the medium (Serway & Jewett, 2006). If the ideal gas is in an 
adiabatic condition we would have (2) as the relation of pressure (p) and density (p) where 
a is a constant and the exponent y is the ratio of specific heats at constant pressure and 
constant volume for the gas (which has the value 1.4 for air) (Fahy, 2001): 

p = ccpv -» dp/dp = Y(P/P) (2) 

Equation (2) does not generally demonstrate a linear relation between pressure and density 
in an ideal gas but in small variations of pressure and density around the equilibrium state, 
p/p can be considered to be constant and we will have: 

[dp/3p] = Y (5r) -» [dp/dp] = K/p (3) 

where [dp/dp] Q denotes small variations around the equilibrium, p and p are the pressure 
and density of the gas at equilibrium and constant K — yp is called the adiabatic bulk 
modulus of the gas (Fahy, 2001). Based on the above discussion the linear stress-strain 
relation in an ideal gas medium can be considered to exist between variations of pressure 
(dp) and variations of density (dp), having an adiabatic process (no loss) and small 
variations of pressure and density around the equilibrium. 

2.2.2 Dissipation mechanisms 

In section 2.2.1 we observed that under three conditions of having an ideal gas with an 
adiabatic process (no loss) and small variations of pressure and density around the 
equilibrium as a result of sound wave, air can be considered a linear lossless medium of sound 
wave propagation. These assumptions are known to be reasonable for audible sound but we 
need to consider their validation for the ultrasound case. Although we can preserve the small 
pressure variations precondition of linearity for ultrasonic speech application, as we will 
observe shortly, the physics of the problem make the assumptions of an adiabatic process and 
ideal gas behaviour of the air for ultrasonic frequencies, to be more of an approximation. 
We need to consider the effects of this approximation i.e. attenuation (heat loss) and also 
deviation of the air from linear state equation (3) of an ideal gas in the frequency range of LF 
ultrasound. These derivations could cause dissipative behaviours in the air medium of 
sound propagation as a result of several phenomena including viscosity, heat conduction 
and relaxation. We will describe each briefly. 

2.2.2.1 Viscosity and heat conduction 

Viscosity is a material property that measures a fluids resistance to deformation. Heat 
conduction on the other hand is the flow of thermal energy through a substance from a 
higher to a lower-temperature region (Licker, 2002). For air, viscosity and heat conduction 
are known to have negligible dispersive effects (section 2.3.4) for sound frequencies below 
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50 MHz (Blackstock, 2000) but these mechanisms cause absorption of sound energy. Their 
effect in an unbounded medium can be considered by introducing a visco-thermal 
absorption coefficient a tv to the time harmonic solution of the wave equation, the amount of 
which demonstrates the necessity of switching to wave equations in thermo-viscous fluids 
for the analysis of waves in frequency range of interest. 

2.2.2.2 Relaxation 

Gases demonstrate a behaviour called relaxation in sound wave propagation. Relaxation 
denotes that there is a time-lag (relaxation delay time) between the initiation of the 
disturbance by the wave and application of this disturbance to the gas which is compared to 
the time a capacitor needs to reach its final voltage value in an RC circuit (Ensminger, 1988). 
This delay could result from several physical phenomena. First the viscosity, second heat 
conduction in the gas from the places which the wave has compressed to the places where 
the wave has rarefacted which will cause the energy of the wave to be distributed in an 
unwanted pattern delaying the energy from returning to the equilibrium. The third and the 
most important case of relaxation in LF ultrasound applications is the molecular relaxation 
resulting from the delays of multi-atomic gas molecules having several modes of 
movement, vibration and rotation and the delay for molecules to be excited in their special 
vibration mode (Crocker, 1998). 

When a new cycle of the wave is applied to the relaxing medium, the delay between the 
previous cycle of the wave disturbance and the resulting response of the medium will 
consume some of the energy of the new cycle, to return the medium to its equilibrium. This 
will cause absorption of the wave energy which depends on the frequency of the wave and the 
amount of the delay. In addition, due to the relative variations of frequency and relaxation 
delay, waves of some frequency can propagate faster than other frequencies. Consequently, 
relaxation in the gases is the physical cause of frequency dependant energy absorption and 
dispersion of the wave. As for this being a reason for dispersion, readers may refer to a 
mathematical discussion in (Bauer, 1965), while for the absorption as a result of relaxation, the 
interesting discussions in (Ingard, 2008) and (Blitz, 1967) should be consulted. 

2.3 Effects of the medium on ultrasound propagation 

Having considered the dispersive mechanisms of a gas for ultrasound frequencies, now we 
can consider the effects of these mechanisms in attenuation and dispersion of ultrasound. 
We will also discuss the case of resonance in the medium of ultrasonic propagation because 
these analyses will finally be applied to the propagation of ultrasound in the vocal tract 
which is a resonant cavity. 

2.3.1 Speed 

The sound speed in a medium (not necessary linear) has been formulated by (Fahy, 2001) as: 

c 2 = dp/ dp (4) 

While a gas medium maintains a linear behaviour as an ideal gas, based on the discussion of 
section 2.2.1, this speed is not a function of frequency and is evaluated according to the 
formula (Blackstock, 2000): 
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c = V^/Po (5) 

If the phase speed of sound propagation in a medium is independent of the frequency as per 
(5), the medium is non-dispersive (Harris & Benenson et al., 2002), and all the events which 
rely on the speed of propagation (such as refraction) will be similar for sound waves across 
the whole frequency range (including ultrasound and audio) in that medium. 

2.3.2 Acoustic impedance 

The concept of acoustic impedance 3 is analogous to electrical impedance and is defined as 
the ratio of acoustic pressure p and the resultant particle velocity u (Harris & Benenson et 
al., 2002). Impedances determine the reflection and refraction of waves over medium 
boundaries. In a homogenous material the acoustic impedance is a material characteristic, so 
it is called characteristic acoustic impedance and is formulated as: 

Z = | = p .c (6) 

Where p is the density of undisturbed medium and c is the speed of sound (The formula is 
same for both solids and fluids when they are homogenous). From (6) it is observed that in a 
non-dispersive material the acoustic impedance is independent of the frequency, so the 
impedance based characteristics (such as reflection coefficients) will be general to the case of 
all sounds in a non-dispersive medium (Harris & Benenson et al., 2002). 

2.3.3 Attenuation 

Attenuation is the loss of the energy of sound beam passing through a material. Attenuation 
can be the result of scattering, diffraction or absorption (Subramanian, 2006). Scattering and 
diffraction losses are not of much concern in the current application of LF ultrasounds in the 
vocal tract so we are going to discuss absorption in more detail. 

The main causes of absorption of energy in gases in ultrasound frequencies are the 
molecular relaxation and visco-thermal effects. Visco-thermal effects introduce a visco- 
thermal absorption coefficient a tv while molecular relaxation introduces several molecular 
coefficients a M . for each of the M t gases in an N gas mixture (like air). The total absorption 
coefficient a is the sum of these values (Blackstock, 2000). 



a = a t 



ZN 
a Mi (7) 

i=l 



a tv is a scalar multiplicand of f 2 , (/ being the frequency of the sound wave) while a M . is a 

f 2 
scalar multiplicand of 2 (f r is the relaxation frequency of the gas 4 ) (Blackstock, 2000). 

The impact of absorption is usually regarded by the value of absorption coefficient. In an 
unbounded medium for the time harmonic analysis of the wave, the role of absorption 
coefficient a would be an exponential multiplicand e~ ar to be multiplied by the lossless 
wave solution where r is the distance of the inspection point from the source. In bounded 



3 The unit for acoustic impedance is Kg/m /s and is called Rayl, named after Lord Rayleigh. 

1 

4 f r = where r is the relaxation time delay of the gas. 

2ttt 
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media we need to switch to damped wave equations to consider the effect of absorption. 
Absorption is usually accompanied by dispersion (Blackstock, 2000). 

2.3.4 Dispersion 

There are several possible causes for dispersion in a gaseous medium among which 
viscosity, heat conduction and relaxation are the most applicable for propagation of 
ultrasound frequencies. It is known that the dispersive effects of viscosity and heat 
conduction in air at frequencies below 50 MHz are negligible (Blackstock, 2000), so the main 
cause of dispersion in lower frequency ultrasound will be molecular relaxation (Blackstock, 
2000). Sound speed in a relaxing gas with standard temperature and pressure is computed 
by (Crocker, 1998): 

C 2 6 (x) 2 T 2 

C 2 1 + € 1 + OJ 2 T 2 V ' 

c is the speed at angular frequency oo = 2nf, e is the relaxation strength and x is relaxation 
time which are constants for a specific gas. c is the low frequency speed of sound in the gas. 
The value cot = 1 occurs at the relaxation frequency f r and the effect of dispersion in 
frequencies around f r is more intense. For example C0 2 introduces dispersion at ultrasonic 
frequencies around 28 kHz (Dean, 1979). 

2.3.5 Resonance 

An important attribute of some sound propagation media is resonance at certain frequencies. 
Resonance is tied closely with the presence of standing waves in a medium. A resonant 
medium for sound waves should first have the possibility of forming standing waves and 
second the capability of frequency selectivity. Standing waves are normally formed as a result 
of interference between two waves travelling in opposite directions. For an interesting 
description of how standing waves are formed in an open-closed end tube as a simplified 
model of vocal tract, readers may refer to (Johnson, 2003). 

The major cause of resonance for sound waves of certain frequencies in a medium is the 
geometric structure of that medium. When the geometry is more suitable for sound waves of 
certain frequencies to be distributed as standing waves in the medium e.g. the medium 
dimensions are wider where the standing wave has a rarefaction and narrower where it has a 
compression point, resonance can happen at that frequency. The resonance frequencies of an 
open/ open and closed/ open tube are a clear example of this (Halliday & Resnick et al., 2004). 
For the case of interest, namely ultrasonic propagation through the vocal tract, we need to 
emphasize that the resonant behaviour of the VT will have one major difference with the 
audible case. In audible frequencies, due to the relatively large wavelength of the sound, 
standing wave patterns establish mainly along the axial length of the tract. But as we move 
toward lower wavelengths, in addition to axial standing waves, cross-modes of resonance 
can be created across the width of the tract, resulting in more complex patterns of resonance. 
Analysis of these cross-modes urges us to consider three dimensional equations for 
ultrasonic wave propagation in the tract while in audible range we normally consider the 
one dimensional wave equation. 
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Now that we have understood the main characteristics of ultrasound and its deviations from 
the general sound category in terms of attenuation and dispersion, we will consider a 
numerical analysis of the impact of these characteristics in LF ultrasound. 

3. Low-frequency ultrasound 

A major application of ultrasound is scanning, both in medical and industrial applications, 
relying upon reflections of the wave by an object (such as a defect in non destructive testing 
or a human fetus in ultra-sonography). When the dimensions of the reflecting object are 
smaller than the wavelength, the wave does not reflect back but scatters as an unfavourable 
wave behaviour. So to detect a defect, one needs to use a wavelength equal or smaller than 
its dimensions e.g. for a defect size of millimetres we need to use a sound wave above MHz 
frequency (Subramanian, 2006). The demand for detecting smaller details moves us out of 
audible range to use higher ultrasound frequencies, limiting the application of LF 
ultrasound to special cases such as cavitation or industrial non destructive testing. 
Low Frequency ultrasound in ultrasonic speech application is considered as a portion of the 
ultrasonic bandwidth, starting from human hearing threshold up to 100 kHz. We will 
discuss the reasons for selection of this portion of the bandwidth shortly. As we will see in 
this section, LF ultrasound has properties which make it a suitable substitute for audible 
excitation of the vocal tract to produce ultrasonic speech. 

The discussion of this section is biased so that the numerical analysis will provide us with an 
insight about the impact of attenuation and dispersion effects of LF ultrasound propagation 
in the vocal tract which we should discuss before being capable of modelling ultrasonic 
speech process as a linear and lossless system. 

We are going to consider attributes of LF ultrasonic propagation in the air, and through the 
air-tissue interface. Soft body tissues and the air in the vocal tract are the regions of interest 
for ultrasonic speech production and both can be considered as homogeneous fluids 
(Zangzebski, 1996). Sound waves in the volumes of fluids are longitudinal (Fahy, 2001) so 
the mode of ultrasound propagation in the vocal tract and soft tissues of our concern will be 
longitudinal. As we will see in this section, high reflection coefficients of the air-tissue 
interface will reflect back most of the ultrasound wave energy over vocal tract walls, so we 
do not need to consider LF propagation through human body tissue. 

3.1 Propagation through air-tissue interface 

As described in (Caruthers, 1977), if the wavelength of the wave is small enough in 
comparison to the dimensions of the boundary of two media, Fermat principle will govern 
and the wave will be reflected with an angle (to the normal) equal to the angle of incidence. 
The reflection coefficient (Crocker, 1998) determines the proportion of energy to be reflected. 
Referring to (Zangzebski, 1996), we observe that the acoustic impedance of the air is too 
small in comparison to other materials of our problem. The reflection coefficient for an air- 
tissue interface (acoustic impedance Z 1 =0.0004* 10 6 Rayls for air and Z 2 =1.71* 10 6 for 
muscle) 5 , is computed to be -0.99 (same value with positive sign for the tissue-air interface) 6 . 



5 Speed of sound is approximated 1600 m/s in muscle and 330 m/s in the air. 

6 The minus value merely indicates the phase difference between the incident and reflected 
signal to be 180 degrees. 
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The value illustrates that ultrasound will almost completely reflect back from an air/ tissue 
or tissue/ air interface. This is expected also by the impedance mismatch effect (Zangzebski, 
1996). 




Frequency (Hz) 
Fig. 1. Variation of the absorption coefficient of the air with frequency 

3.2 Propagation through the air 

In ultrasonic speech applications, the ultrasonic signal entering the vocal tract from the 
transducer has to travel through the air bounded by VT walls. As the exclusive effects of the 
medium on ultrasound, attenuation and dispersion are frequency-dependant, we need to 
have a numerical overview of the significance of these effects on ultrasound propagation in 
the air. 



3.2.1 Attenuation 

The absorption coefficient a was introduced in section 2.3.3 to be a sum of visco-thermal a tv 
and molecular relaxation coefficients. For the air the two major components of oxygen and 
nitrogen have the molecular relaxation coefficients of a N and a . Figure 1 demonstrates the 
variation of value of a (being equal to a tv + cc N + a ) with frequency. As the figure 
demonstrates, this value reaches around 0.1 N p /m in sound frequency of 100 KHz which is 
less than 1 dB/m. 



3.2.2 Dispersion 

As stated in 2.2.1 and 2.3.1, one precondition of linearity for ultrasound propagation in air is 
that the air medium should be an ideal gas in which the speed of sound is independent of 
sound frequency. For frequencies in the ultrasonic range, air deviates from this attribute as a 
result of being composed of dispersive carbon dioxide (CO2) which should be considered in 
the VT due to the higher proportion of CO2 in the exhaled air flow (The percentage of CO2 in 
exhaled air is 4% which is 100 times that in normal air (Zemlin, 1997). This deviation 
initiates at frequencies above 28 kHz (Dean, 1979) and needs to be addressed here in detail. 
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The visco-thermal dispersion of sound in air for frequencies below several hundred MHz, 
depends on the square of the frequency but is negligible for frequencies between 1 Hz and 
50 MHz at STP^ (Blackstock, 2000; Dean, 1979). Thus there remains only molecular 
relaxation dispersion. Among the main components of air (nitrogen, oxygen, carbon dioxide 
and water), nitrogen and oxygen can be considered non-dispersive as the maximum 
variation of sound speed in these two gases with the increase of frequency from zero to 
infinity is only a few centimetres per second (Blackstock, 2000). Water and carbon dioxide 
have effects on variation of sound speed with frequency in the air. Specifically, pure carbon 
dioxide in which the speed of sound may vary about 8m/ s between frequencies of 1kHz 
and 100 kHz (Crocker, 1998). 

Equation (8) demonstrated the dispersion characteristics of the gas, and is shown in figure 2. 
The same figure is reported for air, which illustrates that the dispersive effect of humid air is 
negligible for frequencies up to 5 MHz (Crocker, 1998). 



Fig. 2. Dispersion characteristics of a relaxing gas mixture 

Based on studies of sound propagation in the atmosphere (Dean, 1979), the resulting 
variation of sound speed in air as a mixture of these gases (which obeys figure 2) over 
frequencies up to 5 MHz is in the order of few cm/s (for sound speed of approximately 343 
m/s at STP). Referring to the monotonic pattern of increase of sound speed in (8) and figure 
2, where the maximum speed variation for air at frequencies up to 5 MHz is negligible, and 
considering the percentage of gases other than carbon dioxide in the air, the dispersive 
effects of air can confidently be considered negligible for the dimensions of the vocal tract 
and the frequency range of interest (namely, less than 100 kHz). 

As a conclusion of the preceding discussion, for ultrasonic frequencies of less than 100 kHz, 
and for the dimensions of our problem the air only has the effect of frequency dependant 
attenuation with an absorption coefficient of less than 1 dB/m and can be considered as a 
lossless non-dispersive linear medium in modelling ultrasonic propagation in the vocal 
tract. Linear systems are considered preferential for speech analysis and processing, and so 
we would prefer to limit our application to frequency ranges which can assure a linear 
relationship, if possible. 



7 Standard temperature and pressure. 
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4. Application of LF ultrasound in speech augmentation 

Having described the preliminary basics, we now turn our attention to the application of 
ultrasound in speech augmentation. We will divide these applications into two sets. The 
first set corresponds to applications in which ultrasonic excitation can act as a substitute to 
replace the natural excitation of the human voice production system. In this case, a person 
can speak without any voicing and an ultrasound to audible conversion system can produce 
a final audible sound. In the second set, ultrasonic excitation can be considered to act as a 
supplement to the natural excitation to provide additional data from the vocal tract for 
computational analysis. 

Examples of the former set apply to people who suffer from impairments to their voice box 
and are incapable of producing natural excitations in their VT including laryngectomised 
patients and the voice-rest cases (Pozo, 2004). Another example is where audible speech is 
highly affected by surrounding or background noise and common levels of conversation or 
even high amplitude speech cannot be heard, such as at airports, on the battlefield, or in 
industrial environments (MacLeod, 1987). The other application in this set is when one does 
not wish to be heard in cases of talking in private places or when being heard will disturb 
other applications of a system like dictation in human-computer interfaces of crowded offices. 
For the examples of the second set we may primarily consider ultrasound for providing 
additional data in speech recognition systems aiming to achieve higher levels of robustness. 
As another application in this set, we can mention cases where ultrasound can be 
augmented as an auxiliary excitation to the VT to provide voicing information when 
converting whispered speech to normally phonated speech. In this application, while a 
person whispers, the unvoiced segments of speech are extracted from the whispered signal 
but the voiced segments are reconstructed using the VT resonance data extracted from the 
ultrasonic output of the VT. This special augmentation can be used in whispered speech 
communications over telephone, and speech aids for people who have to speak in whisper 
mode for medical reasons. 



4.1 Ultrasonic speech 

In this chapter the application of LF ultrasonic waves in speech augmentation is termed 
ultrasonic speech. By ultrasonic speech we mean a system which augments an ultrasonic 
excitation to the human voice production mechanism as a substitute or supplement to the 
natural excitation and extracts feature sets from the resulting ultrasonic output to be used in 
several tasks including conversion to the audible speech, speech regeneration, recognition, 
enhancement and communication. The signal which is injected from an ultrasonic 
transducer to the VT via several possible injection points propagates through the tract and 
emits out of the mouth, where it is picked by another transducer and is delivered to the 
processing algorithms in charge of feature extractions in the ultrasonic domain or the 
equivalent audible domain. The set of these extracted features are then delivered as the 
output of the ultrasonic speech system to other modules which may pursue classic tasks of 
speech generation, recognition, and so on. 

The ultrasonic frequency range of this application starts from the higher threshold of human 
hearing up to around 100 kHz. As stated before, this frequency range has some 
characteristics which suit the propagation of ultrasonic waves in the vocal tract to be 
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modelled in linear and lossless acoustic domains. In this domain we can be equipped with 
facilities of linear modelling of the VT behaviour in response to ultrasonic excitation. 

4.2 Previous implementations 

Speech processing science relies heavily on data provided by ultrasonic scanning of the 
position of VT articulators as an indirect contribution of ultrasound to speech processing 
(Kelsey & Minifie et al., 1969). As an example we can mention the data provided by real- 
time ultrasonic monitoring of the tongue (Shawker & Sonies, 2005) to speech processing. In 
direct applications, ultrasonic waves are used directly to produce an ultrasonic speech 
signal which is sought for speech processing features (MacLeod, 1987). Similarly, an audible 
signal modulated by an ultrasonic career in ultrasonic communication (Akerman & Ayers et 
al., 1994), or converted to audible speech as a consequence of the non-linearities of the 
system in ultrasonic hearing (Lenhardt & Skellett et al., 1991). 

These are niche examples of several contributions of ultrasonics to speech processing, yet 
there are few examples of the implementation of low frequency ultrasound in speech 
augmentation (ultrasonic speech). To consider further, let us first review the 
implementations of these methods. 

The history of ultrasonic speech goes as far back as 1987 when MacLeod filed a patent for a 
non audible speech generator system (MacLeod, 1987). The system augmented a series of 
pulses similar to the glottal pulse shape in ultrasonic frequency range of 15 to 105 kHz to the 
vocal tract. MacLeod considered the output at the mouth as being an amplitude modulation 
of the ultrasonic input. He then proposed the idea of passing the output to an ultrasonic 
detector where it was down converted to audible range to pursue a further goal of synthesis 
of artificial speech. He considered the injection transducer to be directly placed on the throat 
or in front of the mouth which was equipped with separate noise and pulse generation 
mechanisms to produce voiced and unvoiced phonemes. 

Based on the classification in the preamble of this section, MacLeod's proposed system was 
a substitutive approach which converted a speaker's silently mouthed words into 
synthesized audible speech. Other later authors mainly considered supplementary 
ultrasonic excitation, mostly for speech recognition. (Tosaya & Sliwa, 2002; 1999) patented a 
system which applied ultrasonic signal injection to the vocal tract to make the task of 
audible voice recognition more robust. Their system was proposed to enhance or replace the 
natural excitation with an artificial excitation for which ultrasound was considered an 
option. The injection points for the artificial excitation were proposed to include: outside 
and within the mouth, nasal passage and on the neck. 

Another instance of ultrasonic speech implementation was proposed by (Lahr, 2002). He 
considered the ultrasonic output of the VT as the third mode of a trimodal voice recognition 
system whose other two modes where audible voice and images of the lips, tongue and the 
teeth. In addition to greater transcription accuracy in the recognition task, the system was 
claimed to be capable of audible speech production when the speaker did not use vocal fold 
vibration and just shaped the VT in positions associated to several different voices. He 
elected to use the neck and mouth as possible injection points of 28 to 100 kHz excitations. 
He also stated that wearing a neck device was usually uncomfortable so he focused on 
signal injection over the lips where the mouth and teeth opening permitted the signal to 
penetrate in the VT. The ultrasonic output of his system was finally demodulated to the 
audible range and used directly as an input channel to a recognition system. 
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Another implementation was reported by (Douglass, 2006), who used ultrasonic excitation 
to add value in improving the reliability of speech recognition. His excitation points were 
below the chin, on the neck, in front, and inside of the mouth. He proposed employing the 
same means of demodulating commonly used in radio broadcasting for the output 
ultrasonic signal. 

4.3 Necessary considerations for implementation 

There are several considerations which are necessary for implementation of an ultrasonic 
speech system. These considerations include, signal injection points, excitation waveforms, 
feature extraction method and hardware setup. 

As stated in section 4.2, in spite of its various applications, ultrasonic speech has been a little 
researched area and there have been few cases of attempts of implementation. One of the 
reasons for unpopularity might be problems associated with signal injection to the vocal 
tract. The choice of injection position has a great impact on system design. Ultrasound, as 
we have observed in section 3.1, reflects back almost totally from the air-tissue interface. 
Another strong reflecting boundary is the bone/ soft tissue interface. The bone is normally 
avoided in ultrasound propagation, because it distorts the ultrasonic beam (Zangzebski, 
1996) (so we will not consider placing the transducer on the jaw or skull bones in this 
chapter). Consequently, injecting the signal through the bone or when the signal is going to 
face an air- tissue interface before entering the VT are not promising options. 
Nevertheless, the task of signal injection is possible via some considerations to prevent or 
compensate for injection problems. Possible injection points introduced by previous 
implementations include the throat, on the neck, against the cheek, in the nasal cavity, inside 
and in front of the mouth. Each of these injection points imposes special considerations to 
fulfil the task of augmentation of an ultrasonic excitation to the VT. 

As an example, for signal injection over the neck skin which has been used by (Lahr, 2002; 
MacLeod, 1987; Tosaya & Sliwa, 2002), the ultrasound wave propagates from the transducer 
to the air gap between the transducer and skin. As we have previously observed, this 
air/ tissue boundary totally reflects the signal back. We can compensate for the effect of the 
reflection by using a coupling gel on the skin to eliminate the air from the transducer/ skin 
interface. The signal entering the skin passes the tissue and encounters another tissue/ air 
boundary before being able to enter the vocal tract where it will almost totally reflect back. 
So to consider signal injection over the neck skin we may need to apply the injection where 
the tissues are relatively thin to minimize reflection effects over the thin boundary. Another 
convenient option is signal injection in front of the mouth. 

Excitation signal waveform design is another task which could simplify and optimize the 
operation of the system. Another brain-storming task is the down conversion of ultrasonic 
output and extraction of features which will be used for the reconstruction or recognition of 
audible speech. Although some of the previously mentioned implementations have 
considered the demodulation of ultrasonic speech to gain the audible equivalent, when the 
resulting converted signal is going to provide features to produce audible speech, the design 
of ultrasonic speech systems will require greater attention. This chapter addresses a solution 
to this issue by mathematically proving the possibility of linear predictive analysis (LP A) of 
ultrasonic speech. LPA is one of the strong feature extraction facilities based on a linear 
source-filter model of speech production. Extension of LPA to the ultrasonic domain will 
significantly simplify processing and analysis requirements in the audible domain. 
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The choice of hardware components in any ultrasonic system is another implementation 
consideration. Transducers are the core of a typical ultrasonic set up, fulfilling the task of 
transmit and receive, but ultrasonic system set up comprises several other hardware 
components including a signal generator to supply input energy to the transmitting 
transducer, and a data acquisition system to capture the signals for analysis. 

5. Human speech production anatomy and physiology 

The human speech production apparatus is well designed for the task of generating, 
modulating, and projecting intelligible sound. Controlled, in part by the Broca nucleus in 
the frontal cortex and Wernicke nucleus in the temporal cortex of the brain, the muscles 
controlling lung exhalation, glottal tension, epiglottis, tongue, throat and lip position, must 
work in concert to create and modulate the sounds that make up language. 
Although speech can be considered as simply as a set of complex waveforms, and indeed 
sinewave speech can be created from simple waveforms (McLoughlin, 2009), it is in reality a 
complex and rich set of auditory symbols differentiated through several production 
mechanisms. These are illustrated in figure 3, and include the following: 

• airflow from the lungs, either restricted, diverted through the nasal passages, 
around the tongue, through the lips or teeth, modulated in speed and intensity, or 
blocked momentarily, as in a plosive sound like /p/. It is the job of the lungs to 
provide the airflow, and to modulate its intensity (although the glottis and lips can 
both be used to block airflow for a time). 

• pitch comes from the vibration of the flap-like vocal cords in the glottis, induced by 
airflow from the lungs. As some muscles in the glottis tauten, the glottal opening 
narrows and the vibration consequently increases in frequency. Pitch not only 
provides the characteristic frequency of our speech, but contributes a lexical 
meaning in several languages, particularly Chinese. Perhaps the most important 
role of pitch, which is similar in many ways to a periodic pulse train, is to resonate 
through the vocal tract. 

• vocal tract geometry dictates the resonance patterns produced by the excitation. A 
pitch train flowing through the VT causes these resonances which affect the 
frequency of the sound exiting the tract in much the same way as most wind 
instruments operate. 

Consider further this analogy with a wind instrument: a trumpet player relies upon a 
mouthpiece which, when blown, acts with the lips to produce a buzzing sound. This takes 
the place of the glottis in the speech production mechanism (and both examples require 
lungs to make the air move in the first place). The annoying buzzing sound from a trumpet 
mouthpiece, when fed through the smooth tubes of a trumpet, results in a beautiful resonant 
horn sound. Pressing or releasing the trumpet valves (keys) selects the tubes that the air 
passes through, resulting in different notes being played. Similarly, the glottal vibration is 
modified by the vocal tract to produce speech sounds. Changing the geometry of the vocal 
tract under muscular control changes the sounds produced in speech (McLoughlin, 2009). 
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Fig. 3. A cut-away diagram of the human speech production mechanism, namely the human 
head (top), along with a block diagram representation below, showing lung excitation 
causing pitch to be produced by the glottis, acted upon by the vocal tract, and emitted from 
the mouth and nose 



In speech, pitch is not present in all sounds: the vowel /a/ is voiced, meaning that it contains 
pitch, whereas the letter /{/ is unvoiced - meaning there is no pitch, so the sound is all lung 
excitation plus vocal tract shape. However all vowels are voiced, as are many consonants. 
In ultrasonic speech production, an ultrasonic pulse-train usually replaces the pitch 
component generated by the glottis. All other articulators remain: the lungs still exhale, and 
provide airflow for the quiet unvoiced sounds (which are around 16dB quieter than voiced 
sounds). The tongue, lips and throat muscles still act together and the human brain can still 
direct the voice production apparatus to form words, as if whispering (which is naturally 
unvoiced). The main difference being that the pulse-producing glottis does not resonate. 
Finally, understanding the speech production mechanism led many researchers to adopt a 
source-filter model for speech. This model separates the sound source (lung and glottis), 
from the filter (vocal tract), and assumes that these two parts are independent, but when 
directed by the brain to act in concert, produce the required sounds. Almost all modern 
speech analysis and processing systems rely heavily upon the source-filter model, and in 
particular assume that the filter part of the model can be represented by a linear polynomial 
function. It is this important relationship that we aim to establish for the case of LF 
ultrasonic speech. 
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6. Modelling ultrasonic speech process 

Linear partial differential equations (PDEs) are the basic descriptors of linear systems, as a 
consequence of allowance to the principle of superposition (Coleman, 2005). Well known for 
benign impulse response and convolutional characteristics, linear time invariant (LTI) 
systems theory has underpinned the source-filter model of speech production for decades. 
The aim of this section is to derive a linear model for the propagation of ultrasonic signals 
through the vocal tract. We have seen in the previous sections that the assumptions of 
lossless propagation and ideal gas behaviour are plausible for small amplitude LF 
ultrasound propagation within the vocal tract. We commence our modelling from basic 
acoustic equations and apply these. 

6.1 Mathematical description of ultrasonic propagation in the VT 

The theory of acoustics stems from four main PDEs based on the conservation of mass, 
momentum and energy and also equations of the state of the medium (Blacks tock, 2000), 
valid in three dimensional space over the frequency range of sound waves (including 
infrasound, audio and ultrasound). These equations are generally not linear but they are 
linearized in acoustics under several simplifying assumptions (Reynolds, 1981) and lead to 
the facilities of the theory of linear acoustics. We have theoretically described these 
assumptions earlier but will now review them mathematically before building on them 
further. 

The first assumption is to consider ultrasonic wave propagation to be an adiabatic (lossless) 
phenomenon. We observed that the main causes of attenuation in ultrasound frequencies in 
a fluid medium are heat conduction, relaxation and viscosity. We then observed in section 
3.2.1 that the effect of this attenuation in the frequency range of our application is negligible. 
So the process could be considered lossless (adiabatic) in which case, the equation of energy 
conservation will not be necessary (Blackstock, 2000). 

The remaining equations are conservation of momentum (9) and mass (10) and equations of 
state of the gas. These equations describe the evolution of pressure p and particle velocity 
vector it as functions of time t and three dimensional coordinates, r — [xy z] . The general 
form of these equations is as stated below (Reynolds, 1981) where p is the density, \i and X 
are viscosity coefficients of the medium and F is the external excitation force. 



P{ d ™/dt + v - ( fi ® **)) = _V P + ( 2 /* + >0 V(V. U)-iiVxVxu + F 



(9) 



3 % t + V.(pu) = (1Q) 



Equation (9) includes the divergence of a dyadic product which is defined as: 



17® u : 



[u ± u 2 u 3 ] 



v 1 u 1 v 1 u 2 v t u 3 
v 2 u 1 v 2 u 2 v 2 u 3 

V 3 U X V 3 U 2 V 3 U 3 



(11) 



where u t is the i th element of the vector u. 

The system (9-10) is completed by the equation of state that gives the pressure as a function 
of the density and temperature. When the flow is adiabatic in a gas, that is, no heat is 
transferred to or from the gas, and is reversible, that is, the flow conditions can return to 
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their original values, the pressure is a function of the density only (Fahy, 2001), and the 
equation of state of the gas reduces to: 

V = P(P) ( 12 ) 

Considering the equation of conservation of momentum (9), with adiabatic and reversible 
wave deformation in the medium, the next assumption is irrotational flow, Vxu = 0. This 
assumption has been somehow challenged by the existence of rotational flows in turbulent 
and jet flows in the classical linear modelling of audible sound propagation in the vocal tract 
during articulation of unvoiced utterances. 

Due to the work of (Lighthill, 1952) and (Goldstein, 1984) the production of turbulent flow 
is governed by nonlinear equations of acoustics but once fully developed, we can describe 
its propagation as irrotational, governed by equations of linear acoustics (Crocker, 2007). We 
have conventionally used this assumption for the audible case, transferring the non-linearity 
of turbulent flow production to the source and dealing with the VT as a linear filter in the 
conventional source-filter modelling of the speech production system (Sinder, 1999). The 
same considerations apply to the ultrasonic range and make the assumption of V x u — a 
plausible statement. 

The next step is to consider the effects of viscosity. Based on the discussions of section 3.2.2 
about negligible dispersive effects of viscosity for frequencies below 50 MHz and referring 
to section 3.2.1 about values of visco-thermal absorption coefficient of the air in the 
frequency range of the current application, we can consider \i and A to be very small, to 
neglect the effects of viscosity for LF ultrasound propagating in the air. We may now rewrite 
(9) in a clearer notation of (13) for each j from 1 to 3 as: 

dUj v-- 3 d(u;U ; -)\ dp _ (13) 



p [-dF + L i=1 ^r) + d^- Fi 



Considering Small disturbances in pressure and density we will have (14, 15) where p , p , 
u are attributes of the medium at equilibrium state which are actually the time averages of 
p, p and u respectively. " Acoustic pressure" p is introduced here then as the small variations 
of pressure around the equilibrium value p . 



P = Po + P; P = Po+P; u = u +u 



(14) 



dp dp du Q 

^= 0; ^= 0; -£= Q ^ 

Assuming the homogeneous (16) medium initially at rest (17): 

Vp = ; Vp = ( 16 ) 

u o = (17) 

And manipulating conditions of (14-17) in (13), the linear equation of conservation of 

acoustic momentum for a lossless homogeneous medium initially at rest is derived for 
ultrasonic propagation inside the vocal tract by (18): 

du (18) 
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For the equation of conservation of mass (10), using the above assumptions of homogeneous 
medium, small disturbances and medium at rest (14-17), we can determine the following: 

^ + p V.it = (19) 

The equation of state for an ideal gas states that: 

V dp 2 (20) 

P dp 

Where c is the speed of sound. The dispersive effects of air medium are discarded in (20) 
based on the discussions of section 3.2.2. Taking the derivative of (20) with respect to time, 
we will have: 

dp^ 2 d_p_ (21) 

dt C dt 

Substituting (21) in (19) we would reach to the conservation of mass equation for ultrasonic 
propagation in the vocal tract: 

1 dp (22) 

We would rewrite (18,22), i.e. lossless linear acoustic equations in (23,24) as the basic 

equations of ultrasound propagation in the vocal tract where p is the acoustic pressure and 

u is the acoustic velocity vector, p is the static mass density of the medium and K is the 

adiabatic bulk modulus of the air: 

du , nns 

Po^ + Vp = F (23) 

dt 

As observed mathematically, the derivation of ultrasonic wave propagation in the vocal 
tract, with the simplifying assumptions which we have described in detail, has led to 
equations (23), (24) which are the general equations of linear acoustics, now applicable for 
ultrasonic propagation through the vocal tract. 

6.2 Vocal tract transfer function for ultrasonic speech 

In our approach to derive a linear model, in this section the VT transfer function is 
determined using the functional transformation method (FTM) which converts the linear 
PDEs to algebraic equations including boundary and initial conditions, similarly to Laplace 
transformation in ordinary PDEs (Rabenstein, 1999). 
Combining (23) and (24) yields the wave equation for p(t, r) and u(t, r): 

——^-V 2 p = ; — — --V 2 u = (25) 

c 2 dt 2 ^ ' c 2 dt 2 v ' 

where r = [x y z] is the three dimensional coordinates vector and c is the speed of sound. 
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For audible sound production, since the cross section of the VT is small compared to the 
wavelength, the wave can propagate along the tract axis and we can model the VT simply as 
a single narrow tube. However the smaller wavelength of ultrasound means the wave can 
propagate across the width of the tract and the resulting cross modes require (25) solving in 
three dimensions. Thus the task of derivation of the three dimensional VT transfer function 
may not be as simple as the one dimensional wave equation for audible sound. We are 
considering the placement of the source in front of the mouth, however the general method 
is applicable to other injection positions. 

Representing VT volume as Q and its boundary as T being comprised of boundaries I\ (the 
glottis), T 2 (VT walls) and T 3 (the mouth), having f(r, t) to be the ultrasonic excitation source 
placed in front of the mouth, the general boundary and initial conditions of ultrasonic wave 
propagation in the VT can be found, with Z(r) being the impedance of the VT and closed 
glottis walls, as: 

p(0,r) = ; reQ; medium initially at rest 

^(0,r) = p (r); reQ (26) 



<»■£') 



„„v + iss>' ,(r ' ,) = rer " r - 



n.Vp(r,t) =f(r,t) reT 3 

for pressure as: 



Defining linear differential operators: D t — —, D r — V 2 — — + — — + — , we can rewrite (25) 



±D t ( P (r,t))-D r ( P (r,t)) = o (27) 

Taking the Laplace transform of (27) and considering the initial conditions of (26), we 
convert differential operator D t (p) to the algebraic form: 

r ^s 2 P(r,s) - ^p (r) - D r (P(r,s)) = (28.a) 

(n " (^ V ) + W) S)P(T ' S) = ° reri ' r 2 (28 ' b) 

n. VP(r, s) = F(r, s) reT 3 (28.c) 

P(r,s)is the Laplace transform of p(r,t) . Next we seek another transform T which can 
convert the spatial differential operator D r to algebraic equations. Lacking a general 
transform similar to the Laplace transform in the spatial domain, the spatial Sturm-Liouville 
transform (SLT) (Rabenstein, 1999) is applied: 

T(P(.r)) = P T (/y = J// Q P(r). K(r, p k )dV ( 29 ) 

The dependence upon Laplace transform parameter (s) is omitted for convenience from this 
point on (so P(r,s) is written as P(r) for instance). The aim is to evaluate the kernel 
function K(r, p k ) so that: 

T(D r {P(r)})= p k 2 T(P(r)) + & (T) (30) 

Where /3 k is a scalar coefficient and O fc (H) is a function which depends on the boundary 
conditions of the problem. To reach this goal, we first multiply (28. a) by K(r, p k ). 
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KQr.PJ {±s 2 P(.r)-±p (r) - D r {P(r)} = o} (31) 

Next we take the integral /// dV , dV is the volume element. 

^s 2 ffS a P(r)K(r,p k ) dV - ±fff a p (r)K (r,fa)dV - J// fl K (r,fa)D r {P(r)} dV = (32) 
Referring to the definition of the SL transform (29), (32) yields: 

^ 2 M&) -^Po T (/« - ffS a K{r,fS k )D r {P(r)}dV = (33) 

Considering D r = V 2 and by Green's theorem (Rabenstein, 1999), the integral in (33) is: 

T{D r {P(r)}} = fjj K(r,p k )D r {P(r)}dV = fjj K(r,p k )V 2 P(r)dV 

= jjj [P(r)V 2 K{r,p k )]dV (34) 

+ [[ K(r,p k )VP(r).ndcr- [[ P(r)VK(r,p k ).nda 



nda is the surface element. Comparing (34), and (30), the first integral in the right hand side 
of (34) should be converted to a multiplicand of T{P(r)} (29). The second integral uses the 
values of VP(r) on the boundary T, which we have by the boundary conditions of (26). The 
last term is unwanted because we do not have the value of {P(r)} over the boundary so we 
define kernel K(r, /3 k ) to fulfil the following requirements as: 

V 2 tf(r,/y =/? fc 2 tf(r,/y (35) 

VK(r,p k ) = reT V } 

Equation (35) is the well known Helmholtz equation (Blackstock, 2000) and its general solution 
relies strongly to the geometry Q. Values of K{r, fi k ), fi k are Eigen functions and Eigen values 
of the operator D r = V 2 (Rabenstein, 1999). We then substitute the results in (33): 

±s 2 P T (p k )-^ PoT (p k )-jjj p k 2 K(r,p k )P(r)dV = jj {K(r,p k )}VP(r).nda (36) 

Referring to the definition of SLT (29) and substituting the values of VP(r) from boundary 
conditions (28.a,b), we may rewrite (36) as: 

(V - h 2 )P T (h)= > 0r (/y " sp ff T ^±K(r,p k )da + ff T K(r,p k )F(r)da = 
^Po T Wk) ~ s Po G gg,r li2 ,/? fc ) + G(F(r),T 3 ,p k ) 

Equation (37), where G{F(r),Y\,Pk) — ff r . K(r,p k )F(r)da, is the general equation relating 

the output P T (Pk) of the VT to the input F(r) and initial and boundary conditions. 
Considering hard walls for both the vocal tract and closed glottis, Z(r) -> oo (based on the 
impedance values of the soft tissue in section 3.1) and Po T (fi k ) = 0, i.e. zero initial conditions, 
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and F(r,s) = F(s) meaning that the ultrasound source has uniform spatial distribution 
pattern, which is a plausible simplification we have: 

G(F(r),r 3 ,/? fe ) =Fff Ts K(r,p k )d<i (38) 

And consequently: 

(c 2 F).JL K(r,B k )da 
Pr(/ y = Jjj *> V ' Pk) (39) 

Since P T (Pk) — T{P(r)}, we need to take the inverse SL transform (Rabenstein, 1999) to reach 
P(r). 

Zoo 1 

fff (40) 

N k = JJJ K\r,p k )dV 
P(r) = P(r, s) is the Laplace transform of p(r, t). Using simplifications of (39), (40) becomes: 



[Z fc=1 W fc { JJ r 



P(r) = Fc 2 [^^{JJ K(r,p k )da}K(r,h) 
And consequently we will reach the transfer function of vocal tract for ultrasonic speech: 



(41) 



H(r) = P -^ = c> [Zl^iJl Kir.MWir.M 



(42) 



H(r)is the three dimensional transfer function of the vocal tract when excited in front of the 
mouth which explicitly is a function of r but in its formation, the integrals were on the 
geometry of volume ft and its boundaries T, so H(r) is strongly relied on the definition of 
the geometry. Thus the three-dimensional wave equation applied to the near-audio 
ultrasonic speech, with several realistic assumptions as described, yields the linear transfer 
function (42). 

6.3 Linear source filter model for ultrasonic speech 

Showing the Laplace transform parameter (s) again - which we had omitted in our 
equations up to the point for simplicity - we recall that H(r) was actually H(r,s), the 
Laplace transform of h(r, t). If sampling time intervals are small enough to consider the VT 
shape pseudo-static, a system with transfer function h{r, t) will be an LTI system, leading to 
a convolutional relation between its output and input as (43). So h(r, t) can be considered as 
a linear time-invariant (LTI) filter for small time intervals and by the benefit of LTI systems, 
the conventional source-filter model of audible speech can be extended to cover ultrasonic 
speech production. 

p(r,t) = h(r,t)*f(t) (43) 

The classical source-filter modelling of VT enjoys independence between source and filter. 
In the case of ultrasonic speech, the source and the filter are intrinsically independent. 
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7. Extension of LPA to the analysis of ultrasonic speech 

In the previous section, linear source filter model of speech production was mathematically 
proven to be valid for ultrasonic speech. Linear source filter modelling of ultrasonic speech 
is the basis of linear predictive analysis as a powerful feature extraction method as will be 
observed in this section. 
The Z transform of h(r, t), can be described as an IIR filter as in (44). 

ZiUfrftQz-' (44) 

W(r ' Z) = l + £?i iai (r) Z -i 

There is a need to inspect the dependence of H(r,z) on coordinates vector r more carefully. 
The VT is a resonant cavity and at ultrasonic frequencies will have cross modes of 
resonance. If the excitation signal is a sine function of the same frequency of one of the 
modes of the resonance, a standing wave of that frequency will form and as a consequence 
of linearity, the output wave at any point, except nodes, will have the same frequency as the 
input. The impulse function is the integral sum of an infinite number of sine waves in the 
time domain. As another consequence of LTI systems, the response of the VT to the impulse 
will be the summation of its output to sine waves of all frequencies including all its 
resonances with different amplitudes. Accordingly although the transfer function would 
have different values in different r, it will have the same set of common poles as the 
resonances of the tract. These common resonances can be calculated with several methods as 
per (Haneda & Makino et al, 1994). 

Linear predictive analysis utilizes the autoregressive (all pole) representation of the transfer 
function of VT and provides the procedures to evaluate the coefficients of the denominator. 
The same procedure can be applied to the Z transform of the VT transfer function in (44) 
which as the transfer function of a minimum phase system, has both poles and zeros inside 
the unit circle and can be represented as an all pole transfer function, with any zeros being 
approximated by additional poles (Rabiner & Schafer, 1978). 

8. Open research questions 

This chapter has presented a mathematical model for ultrasound propagation in the vocal tract 
and has proven the possibility of application of linear predictive analysis to the ultrasonic 
speech. The source-filter model of speech production and LPA are the basic building blocks of 
audible speech processing. Expanding their implementation to ultrasonic speech is the major 
basis of implementation of this technology. Having the findings of this chapter in hand, 
ultrasonic speech can begin to enjoy further research effort to reach a state of maturity. 
For ultrasonic speech, an ultrasound excitation is injected into the vocal tract. The choice of 
optimum excitation point and excitation signal wave-form is a topic for further research. 
Based on the achievements of this chapter, the ultrasonic speech at the output of the mouth 
can be treated as the output of a LTI source-filter model and can be subjected to LPA 
analysis to retrieve a set of common poles of the transfer function. The extracted features, 
converted to a set of parameters, are suitable for production of audible speech. Efficient and 
accurate down-conversion is also a topic of further research which involves the choice of 
suitable deterministic or statistic conversion methods. 
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Finally, as ultrasonic speech involves long term exposure to ultrasound frequencies below 
100 kHz, medical standards in place relating to the health effects of the technology need to 
be assessed and possibly revised as a pre-condition to widespread adoption. 

9. Conclusion 

This chapter has presented ultrasonic speech as a novel application of ultrasound in speech 
augmentation. Ultrasonic speech, operating by replacing the natural excitation in audible 
speech with an LF ultrasonic signal, has applications in speech augmentation for the speech 
rehabilitation and secure communications communities. This chapter has studied the 
requirements in modelling ultrasonic speech as a linear system of sound propagation and 
has proven that LP A, a major tool in the analysis of normal speech, is also extendible to 
ultrasonic speech. 

In pursuing this aim, we first introduced the attributes of ultrasonic propagation in a linear 
lossless gas medium. We observed that if the sound propagation is an adiabatic procedure 
and the gas obeys the ideal gas law and with small disturbances in the medium as a result of 
wave propagation, the gas medium can be considered a linear lossless medium for 
ultrasound propagation. We then discussed deviations of these conditions for ultrasound 
propagation in the air medium. 

Subsequently, LF ultrasound was introduced, and the impacts of the deviations of linear 
acoustic behaviour were numerically analyzed for propagation of low frequency ultrasound in 
the vocal tract. Then we considered the application of LF ultrasound in speech augmentation 
and discussed the aspects of system design which seek more attention. By a review of previous 
implementations, we investigated how they had addressed these aspects including the 
injection points and methods of down-conversion to audible domain. 

Afterwards we considered the physiology and anatomy of the human speech production 
mechanism and how we can substitute the natural excitation with an ultrasonic waveform in 
speech augmentation. We also stated that the ultrasonic excitation could be applied as a 
supplement to natural excitation to provide additional data for speech processing applications. 
The chapter then demonstrated a linear modelling scheme in addition to the fact that speech 
LP A tools can be extended to sound propagation at lower ultrasonic frequencies. Starting 
with basic wave equations, and making several simplifying assumptions such as rigid walls 
for closed glottis and VT, relatively small signal disturbance, and a spatially flat (uniform) 
excitation source , the VT has been shown to be LTI with the transfer function in the form of 
a pole-zero IIR filter. By means of this derivation, the conventional source-filter model was 
proven to be extendable for an ultrasonic speech production system, and thus the powerful 
tools of LPA can be used. 

In this chapter we have tried to bridge from audible speech processing methods to 
ultrasonics by mathematically and physically demonstrating that the extension of principles 
of audible speech processing to the analysis of ultrasonic speech is plausible. This 
significantly simplifies ultrasonic speech processing. The currently neglected area of LF 
ultrasonics research in speech analysis and processing can now be explored with relative 
ease. Further research effort is necessary, and welcomed in this area, as it moves toward 
further maturity and future real-life applications. 
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